Contributing alvarium python SDK implementation to open source project

close project-alvarium#2

* add ReadMe file
* Publish SDK method
* Refactor overall code styling
* Implement Mutate() Method of SDK Interface
* Implement Transit() method for sdk interface
* Implement Create() Method of SDK Interface
* Implement Annotator for Signature Validation (PKI)
* Implement SDK Constructor
* Enable configuration-driven customization of SDK operation
* Implement Annotator for TLS Validation
* Implement Source Annotator
* Implement Annotator for TPM Validation
* Implement Annotator Utils functions
* Implement MQTT StreamProvider
* Fix Annotation json serialization
* Define Annotator interface and factories
* Define StreamProvider interface and the related factories
* Implement ed25519 Sign Provider
* Implement sign provider interface and factory
* Define Initial Annotation Contract Schema
* Implement MD5 HashProvider Type
* Implement SHA256 HashProvider Type
* Define HashProvider interface and its factories
* Initialize the python project structure

Signed-off-by: waleed_badr <[email protected]>

Makefile

@@ -0,0 +1,24 @@
+PYTHON = python3
+SRC_FILES = $(wildcard src/**/*.py)
+
+build: dist 
+
+dist: $(SRC_FILES)
+	$(PYTHON) -m pip install --upgrade build
+	$(PYTHON) -m build
+
+.PHONY: test
+test:
+	$(PYTHON) -m unittest discover -v
+
+.PHONY: clean
+clean:
+	rm -rf dist
+
+.PHONY: install
+install: # only there for testing purposes
+	pip install -r requirements.txt
+
+.PHONY: run-example
+run-example:
+	$(PYTHON) ./tests/example

README.md

@@ -1,2 +1,84 @@
 # alvarium-sdk-python
-Implementation of the Alvarium SDK using Python
+Python implementation of the Project Alvarium SDK
+
+# SDK Interface
+
+The SDK provides a minimal API -- DefaultSdk(), Create(), Mutate(), Transit(), Publish() and Close().
+
+### NewSdk()
+
+```python
+def DefaultSdk(self, annotators: List[Annotator], config: SdkInfo, logger: Logger) --> DefaultSdk
+```
+
+Used to instantiate a new SDK instance with the specified list of annotators.
+
+Takes a list of annotators, a populated configuration and a logger instance. Returns an SDK instance.
+
+### Create()
+
+```python
+def create(self, data: bytes, properties: PropertyBag = None) -> None
+```
+
+Used to register creation of new data with the SDK. Passes data through the SDK instance's list of annotators.
+
+SDK instance method. Parameters include:
+
+
+- data -- The data being created represented as bytes
+
+- properties -- Provide a property bag that may be used by individual annotators
+### Mutate()
+
+```python
+def mutate(self, old_data: bytes, new_data: bytes, properties: PropertyBag = None) -> None
+```
+
+Used to register mutation of existing data with the SDK. Passes data through the SDK instance's list of annotators.
+
+SDK instance method. Parameters include:
+
+- old_data -- The source data item that is being modified, represented as bytes
+
+- new_data -- The new data item resulting from the change, represented as bytes
+
+- properties -- Provide a property bag that may be used by individual annotators
+
+Calling this method will link the old data to the new in a lineage. Specific annotations will be applied to the `new` data element.
+
+### Transit()
+
+```python
+def transit(self, data: bytes, properties: PropertyBag = None) -> None
+```
+
+Used to annotate data that is neither originated or modified but simply handed from one application to another.
+
+SDK instance method. Parameters include:
+
+- data -- The data being handled represented as bytes
+
+- properties -- Provide a property bag that may be used by individual annotators
+
+### Publish()
+
+```python
+def publish(self, data: bytes, properties: PropertyBag = None) -> None
+```
+
+Used to annotate data that is neither originated or modified but **before** being handed to another application.
+
+SDK instance method. Parameters include:
+
+- data -- The data being handled represented as bytes
+
+- properties -- Provide a property bag that may be used by individual annotators
+
+### Close()
+
+```python
+def close(self) -> None
+```
+
+SDK instance method. Ensures clean shutdown of the SDK and associated resources.

pyproject.toml

@@ -0,0 +1,6 @@
+[build-system]
+requires = [
+    "setuptools>=42",
+    "wheel"
+]
+build-backend = "setuptools.build_meta"

requirements.txt

@@ -0,0 +1,2 @@
+ulid-py==1.1.0
+paho-mqtt==1.6.1

setup.cfg

@@ -0,0 +1,23 @@
+[metadata]
+name = alvarium-sdk 
+version = 0.0.1
+author = Alvarium org
+description = Python implementation of the Alvarium sdk 
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://eos2git.cec.lab.emc.com/octo-dcf/alvarium-sdk-python 
+project_urls =
+    Bug Tracker = https://eos2git.cec.lab.emc.com/octo-dcf/alvarium-sdk-python/issues 
+classifiers =
+    Programming Language :: Python :: 3
+    License :: OSI Approved :: Apache License 
+    Operating System :: OS Independent
+
+[options]
+package_dir =
+    = src
+packages = find:
+python_requires = >=3.6
+
+[options.packages.find]
+where = src

src/alvarium/annotators/contracts.py

@@ -0,0 +1,25 @@
+import json
+
+from dataclasses import dataclass
+
+@dataclass
+class Signable:
+    """A data class that holds the seed (data) and signature for this data"""
+
+    seed: str
+    signature: str
+
+    @staticmethod
+    def from_json(data: str):
+        signable_json = json.loads(data)
+        return Signable(seed=str(signable_json["seed"]), signature=str(signable_json["signature"]))
+
+    def to_json(self) -> str:
+        signable_json = {
+            "seed": str(self.seed),
+            "signature": str(self.signature)
+        }
+        return json.dumps(signable_json)
+
+    def __str__(self) -> str:
+        return self.to_json()

src/alvarium/annotators/exceptions.py

@@ -0,0 +1,2 @@
+class AnnotatorException(Exception):
+    """A general exception type to be used by the annotators"""

src/alvarium/annotators/factories.py

@@ -0,0 +1,28 @@
+from alvarium.contracts.config import SdkInfo
+from .interfaces import Annotator
+from alvarium.contracts.annotation import AnnotationType
+from .exceptions import AnnotatorException
+from .mock import MockAnnotator
+from .tpm import TpmAnnotator
+from .pki import PkiAnnotator
+from .source import SourceAnnotator
+from .tls import TlsAnnotator
+
+class AnnotatorFactory():
+    """A factory that provides multiple implementations of the Annotator interface"""
+
+    def get_annotator(self, kind: AnnotationType, sdk_info: SdkInfo) -> Annotator:
+
+        if kind == AnnotationType.MOCK:
+            return MockAnnotator(hash=sdk_info.hash.type, signature=sdk_info.signature, kind=kind)
+        elif kind == AnnotationType.TPM:
+            return TpmAnnotator(hash=sdk_info.hash.type, sign_info=sdk_info.signature)
+        elif kind == AnnotationType.SOURCE:
+            return SourceAnnotator(hash=sdk_info.hash.type, sign_info=sdk_info.signature)
+        elif kind == AnnotationType.TLS:
+            return TlsAnnotator(hash=sdk_info.hash.type, signature=sdk_info.signature)
+        elif kind == AnnotationType.PKI:
+            return PkiAnnotator(hash=sdk_info.hash.type, sign_info=sdk_info.signature)
+        else:
+            raise AnnotatorException("Annotator type is not supported")
+

src/alvarium/annotators/interfaces.py

@@ -0,0 +1,11 @@
+from abc import ABC, abstractmethod
+
+from alvarium.contracts.annotation import Annotation
+from alvarium.utils import PropertyBag
+
+class Annotator(ABC):
+    """A unit responsible for annontating raw data and producing an Annotation object"""
+
+    @abstractmethod
+    def execute(self, data:bytes, ctx: PropertyBag = None) -> Annotation:
+        pass

src/alvarium/annotators/mock.py

@@ -0,0 +1,43 @@
+import socket
+
+from alvarium.contracts.annotation import Annotation, AnnotationType
+from alvarium.hash.contracts import HashType
+from alvarium.hash.factories import HashProviderFactory
+from alvarium.hash.exceptions import HashException
+from alvarium.sign.contracts import SignInfo
+from alvarium.utils import PropertyBag
+from .interfaces import Annotator
+from .exceptions import AnnotatorException
+
+class MockAnnotator(Annotator):
+    """a mock annotator to be used in unit tests"""
+
+    hash: HashType
+    signature: SignInfo
+    kind: AnnotationType
+
+    def __init__(self, hash: HashType, signature: SignInfo, kind: AnnotationType):
+        self.hash = hash
+        self.signature = signature
+        self.kind = kind
+
+    def execute(self, data: bytes, ctx: PropertyBag = None) -> Annotation:
+        hashFactory = HashProviderFactory()
+
+        try:
+            key = hashFactory.get_provider(self.hash).derive(data)
+            host = socket.gethostname()
+            sig = self.signature.public.type.__str__()
+
+            annotation = Annotation(key, self.hash, host, self.kind, sig, True)
+            return annotation
+
+        except HashException as e:
+            raise AnnotatorException("failed to hash data", e)
+
+        except socket.herror as e:
+            raise AnnotatorException("could not get hostname", e)
+
+
+
+

src/alvarium/annotators/pki.py

@@ -0,0 +1,48 @@
+import socket
+
+from alvarium.sign.exceptions import SignException
+from alvarium.sign.factories import SignProviderFactory
+from alvarium.contracts.annotation import Annotation, AnnotationType
+from alvarium.hash.contracts import HashType
+from alvarium.sign.contracts import KeyInfo, SignInfo
+from alvarium.utils import PropertyBag
+from .contracts import Signable
+from .utils import derive_hash, sign_annotation
+from .interfaces import Annotator
+from .exceptions import AnnotatorException
+
+class PkiAnnotator(Annotator):
+
+    def __init__(self, hash: HashType, sign_info: SignInfo) -> None:
+        self.hash = hash
+        self.sign_info = sign_info
+        self.kind = AnnotationType.PKI
+
+    def _verify_signature(self, key: KeyInfo, signable: Signable) -> bool:
+        """ Responsible for verifying the signature, returns true if the verification passed
+            , false otherwise."""
+        try:
+            sign_provider = SignProviderFactory().get_provider(sign_type=key.type)
+        except SignException as e:
+            raise AnnotatorException("cannot get sign provider.", e)
+
+        with open(key.path, 'r') as file:
+            pub_key = file.read()
+            return sign_provider.verify(key=bytes.fromhex(pub_key), 
+                                        content=bytes(signable.seed, 'utf-8'),
+                                        signed=bytes.fromhex(signable.signature))
+
+
+    def execute(self, data: bytes, ctx: PropertyBag = None) -> Annotation:
+        key = derive_hash(hash=self.hash, data=data)
+        host: str = socket.gethostname()
+
+        # create Signable object
+        signable = Signable.from_json(data.decode('utf-8'))
+        is_satisfied: bool = self._verify_signature(key=self.sign_info.public, signable=signable) 
+
+        annotation = Annotation(key=key, host=host, hash=self.hash, kind=self.kind, is_satisfied=is_satisfied)
+
+        signature: str = sign_annotation(key_info=self.sign_info.private, annotation=annotation)
+        annotation.signature = signature
+        return annotation

src/alvarium/annotators/source.py

@@ -0,0 +1,27 @@
+import socket
+
+from alvarium.hash.contracts import HashType
+from alvarium.contracts.annotation import Annotation, AnnotationType
+from alvarium.sign.contracts import SignInfo
+from alvarium.utils import PropertyBag
+from .interfaces import Annotator
+from .utils import derive_hash, sign_annotation
+
+class SourceAnnotator(Annotator):
+    """ A unit used to provide lineage from one version of data to another as a result of
+        change or transformation"""
+
+    def __init__(self, hash: HashType, sign_info: SignInfo) -> None:
+        self.hash = hash
+        self.kind = AnnotationType.SOURCE
+        self.sign_info = sign_info
+
+    def execute(self, data: bytes, ctx: PropertyBag = None) -> Annotation:
+        key: str = derive_hash(hash=self.hash, data=data)
+        host: str = socket.gethostname()
+
+        annotation = Annotation(key=key, hash=self.hash, host=host, kind=self.kind, is_satisfied=True)
+
+        signature: str = sign_annotation(key_info=self.sign_info.private, annotation=annotation)
+        annotation.signature = signature
+        return annotation

src/alvarium/annotators/tls.py

@@ -0,0 +1,49 @@
+import socket
+import ssl
+
+from alvarium.contracts.annotation import Annotation, AnnotationType
+from alvarium.hash.exceptions import HashException
+from alvarium.hash.contracts import HashType
+from alvarium.sign.contracts import SignInfo
+from alvarium.utils import PropertyBag
+from .utils import derive_hash, sign_annotation
+from .interfaces import Annotator
+from .exceptions import AnnotatorException
+
+
+class TlsAnnotator(Annotator):
+
+    hash: HashType
+    signature: SignInfo
+    kind: AnnotationType
+
+    def __init__(self, hash: HashType, signature: SignInfo):
+        self.hash = hash
+        self.signature = signature
+        self.kind = AnnotationType.TLS
+
+    def execute(self, ctx: PropertyBag, data: bytes) -> Annotation:
+        try:
+            key = derive_hash(self.hash, data)
+            is_satisfied = False
+
+            context = ctx.get_property(str(AnnotationType.TLS))
+            if context != None:
+                # If none is returned then there is no error in handshake so tls is satisfied
+                if  type(context) == ssl.SSLSocket:
+                    try:
+                        if context.do_handshake(block=True) == None:
+                            is_satisfied = True
+                    except:
+                        pass
+
+            annotation = Annotation(key= key, hash= self.hash, host= socket.gethostname(), kind= self.kind, is_satisfied= is_satisfied)
+            annotation_signture = sign_annotation(self.signature.private, annotation)
+            annotation.signature = str(annotation_signture)
+            return annotation
+
+        except HashException as e:
+            raise AnnotatorException("failed to hash data", e)
+
+        except socket.herror as e:
+            raise AnnotatorException("could not get hostname", e)