#6 breakdown pt3 - add order_book module (#12)

* refactor repo configs/readme

* add codegen module

* add common module

* remove stale subgraphs/core modules

* add contracts module

* add order_book module

* add order posting example

* add generated api model

* add python-dotenv package

* refactor importing sort

* refactor .env usage and variables requested

* codegen: update reference openapi file for orderbook api codegen

* add E2E_SEPOLIA_TESTING_EOA_PRIVATE_KEY to .env.example

* codegen: regenerate order book models

* api: refactor api base code to handle errors from orderbook api

* orderbook: fix issue with type in order cancellation

* tests(e2e): add e2e order posting test backed by vcr

* ignore .env* files

* api(orderbook): refactor data serialization/deserialization

* tests(e2e): use gnosis mainnet for e2e test

* ci: stop using deprecated version of actions/upload-artifact

* chore: ignore pyright error when instantiating model with string positional param

* chore: remove TODO/WIP comments from README

* ci: ensure E2E_GNOSIS_MAINNET_TESTING_EOA_PRIVATE_KEY is available when running pytest

* chore(ci): revert actions/upload-artifact to v3 to avoid bug when saving conflicting artifacts

actions/upload-artifact#480

---------

Co-authored-by: José Ribeiro <[email protected]>

.env.example

@@ -1,2 +1,2 @@
-USER_ADDRESS=
 PRIVATE_KEY=
+E2E_GNOSIS_MAINNET_TESTING_EOA_PRIVATE_KEY=

.github/workflows/ci.yml

@@ -4,7 +4,7 @@ on:
   workflow_dispatch:
   pull_request:
   push:
-    branches: main 
+    branches: main
   release:
     types: [released]
 
@@ -13,9 +13,7 @@ jobs:
     runs-on: ubuntu-22.04
     strategy:
       matrix:
-        python-version: [
-          "3.10", "3.11", "3.12",
-        ]
+        python-version: ["3.10", "3.11", "3.12"]
     timeout-minutes: 10
     steps:
       - name: Checkout
@@ -49,6 +47,8 @@ jobs:
           poetry run ruff check .
 
       - name: Test
+        env:
+          E2E_GNOSIS_MAINNET_TESTING_EOA_PRIVATE_KEY: ${{ secrets.E2E_GNOSIS_MAINNET_TESTING_EOA_PRIVATE_KEY }}
         run: |
           poetry run pytest tests/
 
@@ -57,7 +57,7 @@ jobs:
           poetry build -f sdist
 
       - name: Archive production artifacts
-        uses: actions/upload-artifact@v2
+        uses: actions/upload-artifact@v3
         with:
           name: wheels
           path: |
@@ -83,4 +83,4 @@ jobs:
       - name: Publish package (prod)
         if: github.event_name == 'release'
         run: |
-          poetry publish --no-interaction -vvv
+          poetry publish --no-interaction -vvv

.gitignore

@@ -124,7 +124,7 @@ celerybeat.pid
 *.sage.py
 
 # Environments
-.env
+.env*
 .venv
 env/
 venv/

Makefile

@@ -6,7 +6,7 @@ web3_codegen:
 	poetry run python -m cow_py.codegen.main
 
 orderbook_codegen:
-	poetry run datamodel-codegen --url="https://raw.githubusercontent.com/cowprotocol/services/v2.245.1/crates/orderbook/openapi.yml" --output cow_py/order_book/generated/model.py --target-python-version 3.12 --output-model-type pydantic_v2.BaseModel --input-file-type openapi
+	poetry run datamodel-codegen --url="https://raw.githubusercontent.com/cowprotocol/services/main/crates/orderbook/openapi.yml" --output cow_py/order_book/generated/model.py --target-python-version 3.12 --output-model-type pydantic_v2.BaseModel --input-file-type openapi
 
 subgraph_codegen:
 	poetry run ariadne-codegen

README.md

@@ -8,7 +8,7 @@ Welcome to the CoW Protocol Python SDK (cow_py), a developer-friendly Python lib
 
 ## 🐄 Features
 
-- Querying the CoW Protocol subgraph.
+- Querying CoW Protocol subgraph.
 - Managing orders on the CoW Protocol.
 - Interacting with CoW Protocol smart contracts.
 - Encoding orders metadata and pinning to CID.
@@ -40,11 +40,11 @@ print(orders)
 
 ## 🐄 Project Structure
 
-- `common/`(WIP): Utilities and configurations, the backbone of the SDK.
+- `common/`: Utilities and configurations, the backbone of the SDK.
 - `contracts/`(TODO): A pasture of Smart contract ABIs for interaction.
-- `order_book/`(TODO): Functions to wrangle orders on the CoW Protocol.
+- `order_book/`: Functions to wrangle orders on the CoW Protocol.
 - `order_signing/`(TODO): Tools for signing and validating orders. Anything inside this module should use higher level modules, and the process of actually signing (ie. calling the web3 function to generate the signature, should be handled in contracts, not here).
-- `subgraph/`(WIP): GraphQL client for querying CoW Protocol's subgraph.
+- `subgraph/`: GraphQL client for querying CoW Protocol's Subgraph.
 - `web3/`: Web3 providers for blockchain interactions.
 
 ## 🐄 How to Use
@@ -110,56 +110,6 @@ data = client.get_data(response)
 pprint(data)
 ```
 
-Or you can leverage `SubgraphClient` to use a custom query and get the results as JSON:
-
-```python
-from pprint import pprint
-from cow_py.subgraph.client import SubgraphClient
-
-url = build_subgraph_url() # Default network is Chain.MAINNET and env SubgraphEnvironment.PRODUCTION
-client = SubgraphClient(url=url)
-
-response = await client.execute(query="""
-            query LastDaysVolume($days: Int!) {
-              dailyTotals(orderBy: timestamp, orderDirection: desc, first: $days) {
-                timestamp
-                volumeUsd
-              }
-            }
-            """, variables=dict(days=2)
-            )
-
-data = client.get_data(response)
-pprint(data)
-```
-
-## 🐄 Development
-
-### 🐄 Tests
-
-Run tests to ensure everything's working:
-
-```bash
-make test # or poetry run pytest
-```
-
-### 🐄 Formatting/Linting
-
-Run the formatter and linter:
-
-```bash
-make format # or ruff check . --fix
-make lint # or ruff format
-```
-
-### 🐄 Codegen
-
-Generate the SDK from the CoW Protocol smart contracts, Subgraph, and Orderbook API:
-
-```bash
-make codegen
-```
-
 ## 🐄 Development
 
 ### 🐄 Tests
@@ -187,82 +137,6 @@ Generate the SDK from the CoW Protocol smart contracts, Subgraph, and Orderbook
 make codegen
 ```
 
-## 🐄 Development
-
-### 🐄 Tests
-
-Run tests to ensure everything's working:
-
-```bash
-make test # or poetry run pytest
-```
-
-### 🐄 Formatting/Linting
-
-Run the formatter and linter:
-
-```bash
-make format # or ruff check . --fix
-make lint # or ruff format
-```
-
-### 🐄 Code Generation
-
-The SDK uses various code generation tools for different components. Here's how to work with them:
-
-#### Full Code Generation
-
-To run all code generation processes:
-
-```bash
-make codegen
-```
-
-This command runs three separate code generation tasks:
-
-1. Web3 Codegen
-2. Orderbook Codegen
-3. Subgraph Codegen
-
-#### Individual Code Generation Tasks
-
-You can also run these tasks individually:
-
-1. Web3 Codegen:
-
-   ```bash
-   make web3_codegen
-   ```
-
-   This runs `python -m cow_py.codegen.main`, which processes the ABIs in the `cow_py/contracts/abi` directory and generates corresponding Python classes.
-
-2. Orderbook Codegen:
-
-   ```bash
-   make orderbook_codegen
-   ```
-
-   This uses `datamodel-codegen` to generate models from the CoW Protocol Orderbook OpenAPI specification.
-
-3. Subgraph Codegen:
-
-   ```bash
-   make subgraph_codegen
-   ```
-
-   This uses `ariadne-codegen` to generate code for interacting with the CoW Protocol subgraph.
-
-#### When to Update Generated Code
-
-You should run the appropriate code generation task when:
-
-1. There are changes to the smart contract ABIs (use `web3_codegen`).
-2. The Orderbook API specification is updated (use `orderbook_codegen`).
-3. The subgraph schema changes (use `subgraph_codegen`).
-4. You modify any of the code generation templates or logic.
-
-It's a good practice to run `make codegen` as part of your development process, especially before committing changes that might affect these generated components.
-
 ## 🐄 Contributing to the Herd
 
 Interested in contributing? Here's how you can help:

cow_py/codegen/abi_handler.py

@@ -39,6 +39,17 @@ def to_python_conventional_name(name: str) -> str:
     return CAMEL_TO_SNAKE_REGEX.sub("_", name).lower()
 
 
+def to_camel_case(name: str) -> str:
+    """Converts a snake_case name to a camelCase name."""
+    name = name.lower()
+    return name[0] + name.title().replace("_", "")[1:]
+
+
+def dict_keys_to_camel_case(d: Dict[str, Any]) -> Dict[str, Any]:
+    """Converts all keys in a dictionary to camelCase."""
+    return {to_camel_case(k): v for k, v in d.items()}
+
+
 def _get_template_file() -> str:
     pkg_files = importlib.resources.files(templates)
     return str(next(x for x in pkg_files.iterdir() if x.suffix == ".hbs"))  # type: ignore

cow_py/common/api/api_base.py

@@ -1,11 +1,20 @@
 from abc import ABC
-from typing import Any, Optional
+import json
+from typing import Any, Dict, List, Optional, Type, TypeVar, Union, get_args
 
 import httpx
 
 from cow_py.common.api.decorators import rate_limitted, with_backoff
+from cow_py.common.api.errors import (
+    ApiResponseError,
+    NetworkError,
+    SerializationError,
+    UnexpectedResponseError,
+)
 from cow_py.common.config import SupportedChainId
+from cow_py.order_book.generated.model import BaseModel
 
+T = TypeVar("T")
 Context = dict[str, Any]
 
 
@@ -42,7 +51,7 @@ async def make_request(self, client, url, method, **request_kwargs):
 
 
 class ResponseAdapter:
-    async def adapt_response(self, _response):
+    def adapt_response(self, _response) -> Dict[str, Any] | str:
         raise NotImplementedError()
 
 
@@ -57,30 +66,99 @@ async def execute(self, client, url, method, **kwargs):
 
 
 class JsonResponseAdapter(ResponseAdapter):
-    def adapt_response(self, response):
-        if response.headers.get("content-type") == "application/json":
-            return response.json()
-        else:
-            return response.text
+    def adapt_response(self, response: httpx.Response) -> Dict[str, Any] | str:
+        try:
+            response.raise_for_status()
+            if response.headers.get("content-type") == "application/json":
+                return response.json()
+            else:
+                return response.text
+        except json.JSONDecodeError as e:
+            raise SerializationError(
+                f"Failed to decode JSON response: {str(e)}", response.text
+            )
+        except httpx.HTTPStatusError as e:
+            raise ApiResponseError(
+                f"HTTP error {e.response.status_code}: {e.response.text}",
+                str(e.response.status_code),
+                e.response,
+            )
 
 
 class ApiBase:
-    """Base class for APIs utilizing configuration and request execution."""
-
     def __init__(self, config: APIConfig):
         self.config = config
+        self.request_strategy = RequestStrategy()
+        self.response_adapter = JsonResponseAdapter()
+        self.request_builder = RequestBuilder(
+            self.request_strategy, self.response_adapter
+        )
+
+    @staticmethod
+    def serialize_model(data: Union[BaseModel, Dict[str, Any]]) -> Dict[str, Any]:
+        if isinstance(data, BaseModel):
+            return json.loads(data.model_dump_json(by_alias=True))
+        elif isinstance(data, dict):
+            return data
+        else:
+            raise ValueError(f"Unsupported type for serialization: {type(data)}")
+
+    @staticmethod
+    def deserialize_model(
+        data: Union[Dict[str, Any], List[Dict[str, Any]], str], model_class: Type[T]
+    ) -> Union[T, List[T]]:
+        if isinstance(data, str):
+            return model_class(data)  # type: ignore
+        if isinstance(data, list):
+            model_class, *_ = get_args(model_class)
+            return [model_class(**item) for item in data]
+        if isinstance(data, dict):
+            return model_class(**data)
+        raise ValueError(f"Unsupported data type for deserialization: {type(data)}")
 
     @with_backoff()
     @rate_limitted()
-    async def _fetch(self, path, method="GET", **kwargs):
+    async def _fetch(
+        self,
+        path: str,
+        method: str = "GET",
+        response_model: Optional[Type[T]] = None,
+        **kwargs,
+    ) -> Union[T, Any]:
         url = self.config.get_base_url() + path
-
-        # remove context_override key used by our backoff decorator
-        clean_kwargs = {k: v for k, v in kwargs.items() if k != "context_override"}
-
-        async with httpx.AsyncClient() as client:
-            builder = RequestBuilder(
-                RequestStrategy(),
-                JsonResponseAdapter(),
+        kwargs = {k: v for k, v in kwargs.items() if k != "context_override"}
+
+        if "json" in kwargs:
+            kwargs["json"] = self.serialize_model(kwargs["json"])
+
+        try:
+            async with httpx.AsyncClient() as client:
+                data = await self.request_builder.execute(client, url, method, **kwargs)
+
+                if isinstance(data, dict) and "errorType" in data:
+                    raise ApiResponseError(
+                        f"API returned an error: {data.get('description', 'No description')}",
+                        data["errorType"],
+                        data,
+                    )
+
+                return (
+                    self.deserialize_model(data, response_model)
+                    if response_model
+                    else data
+                )
+
+        except httpx.NetworkError as e:
+            raise NetworkError(f"Network error occurred: {str(e)}")
+        except httpx.HTTPStatusError as e:
+            if e.response.status_code in (429, 500):
+                raise e
+            raise ApiResponseError(
+                f"HTTP error {e.response.status_code}: {e.response.text}",
+                str(e.response.status_code),
+                e.response,
             )
-            return await builder.execute(client, url, method, **clean_kwargs)
+        except json.JSONDecodeError as e:
+            raise SerializationError(f"Failed to decode JSON response: {str(e)}")
+        except Exception as e:
+            raise UnexpectedResponseError(f"An unexpected error occurred: {str(e)}")