From da1b3159ca5d12494f21a3a68005152e932e0289 Mon Sep 17 00:00:00 2001 From: shadders Date: Sun, 27 Jan 2019 22:32:41 +0000 Subject: [PATCH 1/4] add file, data, meta and gzip protocol specs --- protocols/01-file.md | 31 ++++++++++++++++++++++++ protocols/02-data.md | 31 ++++++++++++++++++++++++ protocols/03-meta.md | 26 ++++++++++++++++++++ protocols/04-gzip.md | 57 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 145 insertions(+) create mode 100644 protocols/01-file.md create mode 100644 protocols/02-data.md create mode 100644 protocols/03-meta.md create mode 100644 protocols/04-gzip.md diff --git a/protocols/01-file.md b/protocols/01-file.md new file mode 100644 index 0000000..a52d00f --- /dev/null +++ b/protocols/01-file.md @@ -0,0 +1,31 @@ +Thanks to Ryan X Charles for the [original spec](https://github.com/bitcoin-sv-specs/op_return/blob/5f051997061ea45873e51a04b494387ad40df05e/protocols/03-file.md) + +# Files in OP_RETURN + +Protocol identifier: `0x66696c65` + +## Specification + +The protocol identfier is a `PUSHDATA` data element `0x04 0x66696c65` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `file`. + +This should be followed by a two further `PUSHDATA` data elements: ` ` + +### Filename + +Contains any data element. However, it is recommended that the filaname be a utf8 character string. Furthermore, it is recommended that it be divided into name and extension like normal files: `.` . Operating systems are already used to dealing with filenames and parsing meaning from them, so we can reuse code from operating systems for this. + + +### Data +Any length of data in any format. However, it is recommended that producers format the file correctly according to the extension. For instance, if your filename is mydocument.pdf then the filedata should be a properly valid PDF file. + +### Example + +An example of a file is as follows: + +``` +OP_RETURN 0x04 0x66696c65 0x09 0x68656c6c6f2e747874 0x06 0x68656c6c6f0a +``` + +This is the file hello.txt consisting of the word "hello" followed by a newline. + + diff --git a/protocols/02-data.md b/protocols/02-data.md new file mode 100644 index 0000000..aa40efe --- /dev/null +++ b/protocols/02-data.md @@ -0,0 +1,31 @@ +# data in OP_RETURN + +Protocol identifier: `0x64617461` + +## Specification + +The protocol identfier is a `PUSHDATA` data element `0x04 0x64617461` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `data`. + +This should be followed by a two further `PUSHDATA` data elements: ` ` + +### Type + +It is recommended that the `type` field contain a utf8 character string describing the data type. A list of well recognised types may be the subject of a later specification. However allowed valuers are not currently part of this specification. + +Example types might include: `bin`, `json`, `jpeg`, `xml`, `html`, `pdf`, `txt`. IANA content types may be supported as well. + + +### Data +Any length of data in any format. However, it is recommended that the data comply with the format specified in the `type` field. + +### Example + +An example of a data item is as follows: + +``` +OP_RETURN 0x04 0x64617461 0x04 6a736f6e 0x0f 0x7b226b6579223a2276616c7565227d +``` + +This is data of type 'json' with a data value of: "{"key":"value"}" + + diff --git a/protocols/03-meta.md b/protocols/03-meta.md new file mode 100644 index 0000000..539620c --- /dev/null +++ b/protocols/03-meta.md @@ -0,0 +1,26 @@ +# data in OP_RETURN + +Protocol identifier: `0x6d657461` + +## Description + +The `meta` protocol is a mechanism of adding meta data to any other protocol by encapsulating it. + +## Specification + +The protocol identfier is a `PUSHDATA` data element `0x04 0x6d657461` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `meta`. + +This should be followed by a further `PUSHDATA` data element: `` which is a a JSON encoded set of the key value pairs. The key value pairs remain unspecified. + +This should be followed by another protocol id and it's subsequent data elemts in their entirety. + +### Example + +We may for example wish to add authorship and encoding data to a text data item. Using the `data` protocol we should do it like this: + +`'meta' '{"author": "alice", "encoding":"utf8"}' 'data' 'txt' 'I am a fish'` + +Which would encode to: +`0x04 0x6d657461 0x26 0x7b22617574686f72223a2022616c696365222c2022656e636f64696e67223a2275746638227d 0x04 0x64617461 0x03 747874 0x0b 0x4920616d20612066697368` + + diff --git a/protocols/04-gzip.md b/protocols/04-gzip.md new file mode 100644 index 0000000..a01d3f6 --- /dev/null +++ b/protocols/04-gzip.md @@ -0,0 +1,57 @@ +# GZIP protocol identifier + +Protocol identifier: `0x677a6970` + +## Specification + +The protocol identfier is a `PUSHDATA` data element `0x04 0x677a6970` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `gzip`. + +The `gzip` protocol identifier is intended to markup another `OP_RETURN` based protocol by adding the simple rule: + +If the `gzip` identifier is present the following `PUSHDATA` operation contains gzip encoded data. The only exception is the very first `gzip` identifier which is used to designate that the gzip handler should parse all the `PUSHDATA` following according to the above rule. To push a single gzipped data element the protocol identifier should be pushed twice followed by the gzipped data. + +The second `PUSHDATA`following the gzip protocol identfier should also be interpreted as a protocol identfier. In order to be interpreted according to the original protocol any data elements marked as gzipped should be decompressed and the gzip identifiers stripped out. + +This protocol is stream friendly and can be implemented as a pipeline of stream handlers: `gzip_handler -> sub_handler` + +### Examples + +##### Gzip a single data element: + +` ` + +Which should be returned from the gzip handler as: + +`` + + +##### Compressed FILE data but uncompressed FILEAME: + +`'gzip' 'file' 'gzip' ` + +This should be passed to the gzip handler which should output: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +##### Compressed FILE and FILENAME: + +`'gzip' 'file' 'gzip' 'gzip' ` + +This should be passed to the gzip handler which should output: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +##### Compressed DATA but uncompressed TYPE: + +`'gzip' 'data' 'gzip' ` + +This should be passed to the gzip handler which should output: + +`'data' ` + +Which is then passed to the `data` protocol handler. + From 1041bae2307590fe926e60e91959734d2bf1984a Mon Sep 17 00:00:00 2001 From: shadders Date: Sun, 27 Jan 2019 22:38:04 +0000 Subject: [PATCH 2/4] add file, data, meta and gzip protocol specs - update protocol_ids.csv --- protocol_ids.csv | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/protocol_ids.csv b/protocol_ids.csv index 63ef6a1..9b063af 100644 --- a/protocol_ids.csv +++ b/protocol_ids.csv @@ -4,6 +4,10 @@ Prefix,DisplayName,Authors,BitcoinAddress,SpecificationUrl,TxidRedirectUrl 0x00584350,Counterparty Cash,Counterparty Cash Association (CCA),17YhxKgKunLjvi7HB1ENGmeLwPFiaxb74V,https://counterparty.cash, 0x00746C6B,Keyport,Keyport,,https://keyport.io, 0x02446365,SatoshiDICE,Jon Bestman,1DiceuELb5GZktc3CMEv868DMtU3B5957x,https://satoshidice.com, +0x64617461,data,Steve Shadders,,, +0x66696c65,file,Ryan X Charles,,, +0x677a6970,gzip,Steve Shadders,,, +0x6d657461,meta,Steve Shadders,,, +0x6d6f6e6579627574746f6e2e636f6d,Money Button,Ryan X. Charles,,https://www.moneybutton.com, 0xac1eed88,Miner ID,Steve Shadders,,, 0xff000000,extended protocol id mask,,,, -0x6d6f6e6579627574746f6e2e636f6d,Money Button,Ryan X. Charles,,https://www.moneybutton.com From 54f0fbbe7a846e8d2672faa39dd934ce31aae21f Mon Sep 17 00:00:00 2001 From: shadders Date: Mon, 28 Jan 2019 23:37:57 +0000 Subject: [PATCH 3/4] Guideline to come. Modify GZIP and add generic transformation and marup descriptions. --- protocols/04-gzip.md | 57 --------------------- protocols/1.0-general-protocols.md | 0 protocols/{01-file.md => 1.1-file.md} | 0 protocols/{02-data.md => 1.2-data.md} | 0 protocols/2.0-markup-protocols.md | 5 ++ protocols/{03-meta.md => 2.1-meta.md} | 0 protocols/3.0-transformation-protocols.md | 58 ++++++++++++++++++++++ protocols/3.1-gzip.md | 60 +++++++++++++++++++++++ 8 files changed, 123 insertions(+), 57 deletions(-) delete mode 100644 protocols/04-gzip.md create mode 100644 protocols/1.0-general-protocols.md rename protocols/{01-file.md => 1.1-file.md} (100%) rename protocols/{02-data.md => 1.2-data.md} (100%) create mode 100644 protocols/2.0-markup-protocols.md rename protocols/{03-meta.md => 2.1-meta.md} (100%) create mode 100644 protocols/3.0-transformation-protocols.md create mode 100644 protocols/3.1-gzip.md diff --git a/protocols/04-gzip.md b/protocols/04-gzip.md deleted file mode 100644 index a01d3f6..0000000 --- a/protocols/04-gzip.md +++ /dev/null @@ -1,57 +0,0 @@ -# GZIP protocol identifier - -Protocol identifier: `0x677a6970` - -## Specification - -The protocol identfier is a `PUSHDATA` data element `0x04 0x677a6970` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `gzip`. - -The `gzip` protocol identifier is intended to markup another `OP_RETURN` based protocol by adding the simple rule: - -If the `gzip` identifier is present the following `PUSHDATA` operation contains gzip encoded data. The only exception is the very first `gzip` identifier which is used to designate that the gzip handler should parse all the `PUSHDATA` following according to the above rule. To push a single gzipped data element the protocol identifier should be pushed twice followed by the gzipped data. - -The second `PUSHDATA`following the gzip protocol identfier should also be interpreted as a protocol identfier. In order to be interpreted according to the original protocol any data elements marked as gzipped should be decompressed and the gzip identifiers stripped out. - -This protocol is stream friendly and can be implemented as a pipeline of stream handlers: `gzip_handler -> sub_handler` - -### Examples - -##### Gzip a single data element: - -` ` - -Which should be returned from the gzip handler as: - -`` - - -##### Compressed FILE data but uncompressed FILEAME: - -`'gzip' 'file' 'gzip' ` - -This should be passed to the gzip handler which should output: - -`'file' ` - -Which is then passed to the `file` protocol handler. - -##### Compressed FILE and FILENAME: - -`'gzip' 'file' 'gzip' 'gzip' ` - -This should be passed to the gzip handler which should output: - -`'file' ` - -Which is then passed to the `file` protocol handler. - -##### Compressed DATA but uncompressed TYPE: - -`'gzip' 'data' 'gzip' ` - -This should be passed to the gzip handler which should output: - -`'data' ` - -Which is then passed to the `data` protocol handler. - diff --git a/protocols/1.0-general-protocols.md b/protocols/1.0-general-protocols.md new file mode 100644 index 0000000..e69de29 diff --git a/protocols/01-file.md b/protocols/1.1-file.md similarity index 100% rename from protocols/01-file.md rename to protocols/1.1-file.md diff --git a/protocols/02-data.md b/protocols/1.2-data.md similarity index 100% rename from protocols/02-data.md rename to protocols/1.2-data.md diff --git a/protocols/2.0-markup-protocols.md b/protocols/2.0-markup-protocols.md new file mode 100644 index 0000000..1cb01ea --- /dev/null +++ b/protocols/2.0-markup-protocols.md @@ -0,0 +1,5 @@ +# Markup protocols + +Markup protocols are a family of protocols intended to encapsulate other protocols whilst marking them up with additional data. The general pattern is demonstrated by the `'meta'` protocol which wraps any other protocol with JSON key value pairs as metadata. + + diff --git a/protocols/03-meta.md b/protocols/2.1-meta.md similarity index 100% rename from protocols/03-meta.md rename to protocols/2.1-meta.md diff --git a/protocols/3.0-transformation-protocols.md b/protocols/3.0-transformation-protocols.md new file mode 100644 index 0000000..efd20a4 --- /dev/null +++ b/protocols/3.0-transformation-protocols.md @@ -0,0 +1,58 @@ +# Transformation protocols + +Transformation protocols are intended to modify a data element to produce another valid protocol. The general pattern is that a protocol is encapsulated within the transformation protocol and once the transformation handler has complete the output should be another valid protocol. Two examples of this might be `'gzip'` and `'encrypt'`. + +# Pattern + +Transformation protocols should be able to specify their own parameter patterns. The number of parameters implicitly specifies the begin index of the data element they are operating on, that is, the one immediately following the last parameter. For example if the fictitious protocol `'decompress'` took one parameter `'` The parameter is the 2nd data element in the `OP_RETURN` and the data to be transformed begins at the 3rd element. For optional parameters positional integrity can be maintained by using the null data element for optional parameters you don't wish to specify. A parameter may be another transformation protocol id such that transformations may be nested (e.g. `'tar' 'gzip'`) + +Taking `'gzip'` as an example there are two modes of operation possible. + +1/ If the transformation protocol and it's parameters are followed by only a single data element it is expected to apply a transformation producing an array of bytes that should then be parsed for a sequence of `PUSHDATA` data elements. The output of the transformation protocol is this newly parsed sequence of data elements. + +2/ If the transformation protocol and it's parameters are followed by more than one data element it is expected the data elements will be a sequence of pairs. Each pair is a boolean flag `OP_0` (`0x00`) or `OP_1` (`0x01`), followed by a another data element. The boolean flag indicate with to apply the transformation to that data element. The ouput of the transformation should be sequence of optionally transformed data elements excluding the boolean flags. + +In both case the resulting sequence should be treated as a protocol itself with the first data element expected to be a protocol identifier which indicates which handler to pass the data to. + +##Example + +### Transformation of a single data element + +##### Gzip a single data element: + +`'gzip' ` + +Which should be returned from the gzip handler as: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +### Transformation of multiple data elements + +##### Compressed FILE data but uncompressed FILEAME: + +`'gzip' 0 'file' 0 1 ` + +This should be passed to the gzip handler which should output: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +### Nested transformations + +Using the fictitious transformation protocol `'tar'` which concatenates multiple `'file'` elements into a single output: + +`'gzip' 'tar' ` + +First the `'gzip'` transformation handler will decompress the data element yielding: + +`'tar' ` + +The `'tar'` handler then transforms the output of `'gzip'` into a list of embedded file protocol elements giving: + +`'file' ` `'file' ` + + + diff --git a/protocols/3.1-gzip.md b/protocols/3.1-gzip.md new file mode 100644 index 0000000..b0aea00 --- /dev/null +++ b/protocols/3.1-gzip.md @@ -0,0 +1,60 @@ +# GZIP protocol identifier + +Protocol identifier: `0x677a6970` + +GZIP is a transformation protocol intended to encapsulate another protocol. + +## Specification + +The protocol identfier is a `PUSHDATA` data element `0x04 0x677a6970` which is the 1 byte length prefix followed by the 4 byte identifier which translates to the ASCII string `gzip`. + +The `gzip` protocol identifier is intended to transform another `OP_RETURN` based protocol by adding two simple rules: + +If the `gzip` identifier is present and only one `PUSHDATA` element follows, that element contains gzipped data. It should be ungzipped then the array of bytes should be parsed for `PUSHDATA` elements then treated as another protocol with the first element being a protocol identier. + +If the `gzip` identifier is present and more than one `PUSHDATA` element follows then it should be assumed that each data element is prefixed with a single byte flag indicating whether the following data element is gzipped. `OP_0` (`0x00`) indicates it is not, `OP_1` (`0x01`) indicates that it is. The `gzip` handler should decompress any compressed elements and strip out the flag elements. Returning a decompressed series of `PUSHDATA` elements. This sequence should then be interpreted as a protocol indicated by the first of these data elements as a protocol identifier. + +This protocol is stream friendly and can be implemented as a pipeline of stream handlers: `gzip_handler -> sub_handler` + +### Examples + +##### Gzip a single data element: + +`'gzip' ` + +Which should be returned from the gzip handler as: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +##### Compressed FILE data but uncompressed FILEAME: + +`'gzip' 0 'file' 0 1 ` + +This should be passed to the gzip handler which should output: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +##### Compressed FILE and FILENAME: + +`'gzip' 0 'file' 1 1 ` + +This should be passed to the gzip handler which should output: + +`'file' ` + +Which is then passed to the `file` protocol handler. + +##### Compressed DATA but uncompressed TYPE: + +`'gzip' 0 'data' 0 1 ` + +This should be passed to the gzip handler which should output: + +`'data' ` + +Which is then passed to the `data` protocol handler. + From 37c79b03515b2f360a5e5f3d32b0b1014533f4a8 Mon Sep 17 00:00:00 2001 From: "Ryan X. Charles" Date: Mon, 28 Jan 2019 16:07:31 -0800 Subject: [PATCH 4/4] fix my own name so i get credit in the commit history --- protocol_ids.csv | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/protocol_ids.csv b/protocol_ids.csv index 9b063af..400ec19 100644 --- a/protocol_ids.csv +++ b/protocol_ids.csv @@ -5,7 +5,7 @@ Prefix,DisplayName,Authors,BitcoinAddress,SpecificationUrl,TxidRedirectUrl 0x00746C6B,Keyport,Keyport,,https://keyport.io, 0x02446365,SatoshiDICE,Jon Bestman,1DiceuELb5GZktc3CMEv868DMtU3B5957x,https://satoshidice.com, 0x64617461,data,Steve Shadders,,, -0x66696c65,file,Ryan X Charles,,, +0x66696c65,file,Ryan X. Charles,,, 0x677a6970,gzip,Steve Shadders,,, 0x6d657461,meta,Steve Shadders,,, 0x6d6f6e6579627574746f6e2e636f6d,Money Button,Ryan X. Charles,,https://www.moneybutton.com,