Print token costs after each message

[materemias]

eg aider prints after each answer:

Tokens: 4.0k sent, 264 received.
Cost: $0.02 message, $0.05 session. 

something like this would be beneficial for gptme as well

[ErikBjare]

We have the /tokens command which prints the input costs for the session.

Will consider adding a feature to include the output cost too.

[Kamalesh-Seervi]

can I work on this

[ErikBjare]

@Kamalesh-Seervi Sure, give it a shot! :)

[Kamalesh-Seervi]

I have installation issue I use python venv the installation steps isn't smooth

[Kamalesh-Seervi]

Installing dependencies from lock file

Package operations: 106 installs, 0 updates, 0 removals

  - Installing certifi (2024.8.30)
  - Installing charset-normalizer (3.4.0)
  - Installing idna (3.10)
  - Installing markupsafe (3.0.1)
  - Installing urllib3 (2.2.3)
  - Installing alabaster (1.0.0)
  - Installing fsspec (2024.9.0): Pending...
  - Installing fsspec (2024.9.0): Installing...
  - Installing fsspec (2024.9.0)
  - Installing h11 (0.14.0)
  - Installing docutils (0.21.2)
  - Installing filelock (3.16.1)
  - Installing imagesize (1.4.1)
  - Installing babel (2.16.0)
  - Installing jinja2 (3.1.4)
  - Installing mdurl (0.1.2)
  - Installing packaging (24.1)
  - Installing pygments (2.18.0)
  - Installing pyyaml (6.0.2)
  - Installing requests (2.32.3)
  - Installing six (1.16.0)
  - Installing sniffio (1.3.1)
  - Installing snowballstemmer (2.2.0)
  - Installing soupsieve (2.6)
  - Installing sphinxcontrib-applehelp (2.0.0)
  - Installing sphinxcontrib-devhelp (2.0.0)
  - Installing sphinxcontrib-htmlhelp (2.1.0)
  - Installing sphinxcontrib-jsmath (1.0.1)
  - Installing sphinxcontrib-qthelp (2.0.0)
  - Installing sphinxcontrib-serializinghtml (2.0.0)
  - Installing tqdm (4.66.5)
  - Installing typing-extensions (4.12.2)
  - Installing accessible-pygments (0.0.5)
  - Installing annotated-types (0.7.0)
  - Installing anyio (4.6.2.post1)
  - Installing asttokens (2.4.1)
  - Installing beautifulsoup4 (4.12.3)
  - Installing markdown-it-py (3.0.0)
  - Installing click (8.1.7)
  - Installing distlib (0.3.9)
  - Installing executing (2.1.0)
  - Installing httpcore (1.0.6)
  - Installing huggingface-hub (0.25.2)
  - Installing iniconfig (2.0.0)
  - Installing parso (0.8.4)
  - Installing platformdirs (4.3.6)
  - Installing pluggy (1.5.0)
  - Installing ptyprocess (0.7.0)
  - Installing pure-eval (0.2.3)
  - Installing pydantic-core (2.23.4)
  - Installing sphinx (8.1.3)
  - Installing traitlets (5.14.3)
  - Installing types-html5lib (1.1.11.20240806)
  - Installing wcwidth (0.2.13)
  - Installing cfgv (3.4.0)
  - Installing httpx (0.27.2)
  - Installing coverage (7.6.3)
  - Installing cssselect (1.2.0)
  - Installing identify (2.6.1)
  - Installing decorator (5.1.1)
  - Installing distro (1.9.0)
  - Installing execnet (2.1.1)
  - Installing jedi (0.19.1)
  - Installing jiter (0.6.1)
  - Installing gprof2dot (2024.6.6)
  - Installing greenlet (3.0.3): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  running bdist_wheel
  running build
  running build_py
  creating build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/__init__.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  creating build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/__init__.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  creating build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_version.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_weakref.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_slp_switch.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_gc.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/leakcheck.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_switch_two_greenlets.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_generator.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_greenlet_trash.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_throw.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_tracing.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_cpp_exception.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_switch_three_greenlets.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_cpp.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_contextvars.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_greenlet.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_extension_interface.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_clearing_run_switches.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/__init__.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_switch_three_greenlets2.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/fail_initialstub_already_started.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_generator_nested.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_stack_saved.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/test_leaks.py -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  running egg_info
  writing src/greenlet.egg-info/PKG-INFO
  writing dependency_links to src/greenlet.egg-info/dependency_links.txt
  writing requirements to src/greenlet.egg-info/requires.txt
  writing top-level names to src/greenlet.egg-info/top_level.txt
  reading manifest file 'src/greenlet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files found matching 'benchmarks/*.json'
  no previously-included directories found matching 'docs/_build'
  warning: no files found matching '*.py' under directory 'appveyor'
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.pyd' found anywhere in distribution
  warning: no previously-included files matching '*.so' found anywhere in distribution
  warning: no previously-included files matching '.coverage' found anywhere in distribution
  adding license file 'LICENSE'
  adding license file 'LICENSE.PSF'
  adding license file 'AUTHORS'
  writing manifest file 'src/greenlet.egg-info/SOURCES.txt'
  copying src/greenlet/TBrokenGreenlet.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TExceptionState.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TGreenlet.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TGreenletGlobals.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TMainGreenlet.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TPythonState.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TStackState.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TThreadStateDestroy.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/TUserGreenlet.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_allocator.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_compiler_compat.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_cpython_add_pending.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_cpython_compat.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_exceptions.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_greenlet.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_internal.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_refs.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_slp_switch.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_thread_state.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_thread_state_dict_cleanup.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/greenlet_thread_support.hpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/slp_platformselect.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet
  copying src/greenlet/platform/setup_switch_x64_masm.cmd -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_aarch64_gcc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_alpha_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_amd64_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_arm32_gcc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_arm32_ios.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_arm64_masm.asm -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_arm64_masm.obj -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_arm64_msvc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_csky_gcc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_loongarch64_linux.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_m68k_gcc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_mips_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc64_aix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc64_linux.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc_aix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc_linux.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc_macosx.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_ppc_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_riscv_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_s390_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_sparc_sun_gcc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x32_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x64_masm.asm -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x64_masm.obj -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x64_msvc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x86_msvc.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/platform/switch_x86_unix.h -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/platform
  copying src/greenlet/tests/_test_extension.c -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  copying src/greenlet/tests/_test_extension_cpp.cpp -> build/lib.macosx-15.0-arm64-cpython-313/greenlet/tests
  running build_ext
  building 'greenlet._greenlet' extension
  creating build/temp.macosx-15.0-arm64-cpython-313/src/greenlet
  clang++ -fno-strict-overflow -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -O3 -Wall -I/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmpuxzb9g9b/.venv/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13 -c src/greenlet/greenlet.cpp -o build/temp.macosx-15.0-arm64-cpython-313/src/greenlet/greenlet.o --std=gnu++11
  In file included from src/greenlet/greenlet.cpp:19:
  In file included from src/greenlet/greenlet_internal.hpp:20:
  In file included from src/greenlet/greenlet_greenlet.hpp:26:
  /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_frame.h:8:4: error: "this header requires Py_BUILD_CORE define"
      8 | #  error "this header requires Py_BUILD_CORE define"
        |    ^
  In file included from src/greenlet/greenlet.cpp:19:
  In file included from src/greenlet/greenlet_internal.hpp:20:
  In file included from src/greenlet/greenlet_greenlet.hpp:26:
  In file included from /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_frame.h:13:
  /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_code.h:8:4: error: "this header requires Py_BUILD_CORE define"
      8 | #  error "this header requires Py_BUILD_CORE define"
        |    ^
  In file included from src/greenlet/greenlet.cpp:19:
  In file included from src/greenlet/greenlet_internal.hpp:20:
  In file included from src/greenlet/greenlet_greenlet.hpp:26:
  In file included from /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_frame.h:13:
  In file included from /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_code.h:11:
  /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_lock.h:13:4: error: "this header requires Py_BUILD_CORE define"
     13 | #  error "this header requires Py_BUILD_CORE define"
        |    ^
  In file included from src/greenlet/greenlet.cpp:19:
  In file included from src/greenlet/greenlet_internal.hpp:20:
  In file included from src/greenlet/greenlet_greenlet.hpp:26:
  In file included from /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_frame.h:13:
  In file included from /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_code.h:12:
  /opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.13/include/python3.13/internal/pycore_backoff.h:9:4: error: "this header requires Py_BUILD_CORE define"
      9 | #  error "this header requires Py_BUILD_CORE define"
        |    ^
  In file included from src/greenlet/greenlet.cpp:19:
  In file included from src/greenlet/greenlet_internal.hpp:20:
  src/greenlet/greenlet_greenlet.hpp:104:9: error: unknown type name '_PyCFrame'
    104 |         _PyCFrame* cframe;
        |         ^
  src/greenlet/greenlet_greenlet.hpp:147:29: error: unknown type name '_PyCFrame'
    147 |         void set_new_cframe(_PyCFrame& frame) noexcept;
        |                             ^
  src/greenlet/greenlet_greenlet.hpp:660:17: error: cannot initialize object parameter of type 'const greenlet::Greenlet' with an expression of type 'const greenlet::UserGreenlet'
    660 |             if (this->started() || !this->_run_callable) {
        |                 ^~~~
  In file included from src/greenlet/greenlet.cpp:32:
  src/greenlet/TMainGreenlet.cpp:48:5: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
     48 |     this->tp_clear();
        |     ^~~~
  src/greenlet/TMainGreenlet.cpp:93:9: error: cannot initialize object parameter of type 'const greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
     93 |         this->check_switch_allowed();
        |         ^~~~
  src/greenlet/TMainGreenlet.cpp:96:9: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
     96 |         this->release_args();
        |         ^~~~
  src/greenlet/TMainGreenlet.cpp:100:32: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
    100 |     switchstack_result_t err = this->g_switchstack();
        |                                ^~~~
  src/greenlet/TMainGreenlet.cpp:104:16: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
    104 |         return this->on_switchstack_or_initialstub_failure(
        |                ^~~~
  src/greenlet/TMainGreenlet.cpp:125:22: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::MainGreenlet'
    125 |     return Greenlet::tp_traverse(visit, arg);
        |                      ^~~~~~~~~~~
  In file included from src/greenlet/greenlet.cpp:33:
  src/greenlet/TUserGreenlet.cpp:69:9: error: cannot initialize object parameter of type 'const greenlet::Greenlet' with an expression of type 'const greenlet::UserGreenlet'
     69 |     if (this->started()) {
        |         ^~~~
  src/greenlet/TUserGreenlet.cpp:100:22: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::UserGreenlet'
    100 |     return Greenlet::throw_GreenletExit_during_dealloc(current_thread_state);
        |                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/greenlet/TUserGreenlet.cpp:129:9: error: cannot initialize object parameter of type 'const greenlet::Greenlet' with an expression of type 'greenlet::UserGreenlet'
    129 |         this->check_switch_allowed();
        |         ^~~~
  src/greenlet/TUserGreenlet.cpp:132:9: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::UserGreenlet'
    132 |         this->release_args();
        |         ^~~~
  src/greenlet/TUserGreenlet.cpp:146:15: error: cannot initialize a variable of type 'Greenlet *' with an rvalue of type 'greenlet::UserGreenlet *'
    146 |     Greenlet* target = this;
        |               ^        ~~~~
  src/greenlet/TUserGreenlet.cpp:158:36: error: cannot initialize object parameter of type 'greenlet::Greenlet' with an expression of type 'greenlet::UserGreenlet'
    158 |                 target->args() <<= this->args();
        |                                    ^~~~
  fatal error: too many errors emitted, stopping now [-ferror-limit=]
  20 errors generated.
  error: command '/usr/bin/clang++' failed with exit code 1
  

  at /opt/homebrew/Cellar/poetry/1.8.4/libexec/lib/python3.13/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with greenlet (3.0.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "greenlet (==3.0.3)"'.

  - Installing matplotlib-inline (0.1.7)
  - Installing mdit-py-plugins (0.4.2)
  - Installing mypy-extensions (1.0.0)
  - Installing nodeenv (1.9.1)
  - Installing pexpect (4.9.0)
  - Installing prompt-toolkit (3.0.48)
  - Installing pydantic (2.9.2)
  - Installing pydata-sphinx-theme (0.15.4)
  - Installing pytest (8.3.3)
  - Installing python-dotenv (1.0.1)
  - Installing regex (2024.9.11)
  - Installing stack-data (0.6.3)
  - Installing tokenizers (0.20.1): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  Running `maturin pep517 build-wheel -i /private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmplqbh9ypf/.venv/bin/python --compatibility off`
  🍹 Building a mixed python/rust project
  🔗 Found pyo3 bindings
  🐍 Found CPython 3.13 at /private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmplqbh9ypf/.venv/bin/python
  📡 Using build options features, bindings from pyproject.toml
  💻 Using `MACOSX_DEPLOYMENT_TARGET=11.0` for aarch64-apple-darwin by default
     Compiling proc-macro2 v1.0.86
     Compiling unicode-ident v1.0.13
     Compiling autocfg v1.4.0
     Compiling libc v0.2.159
     Compiling target-lexicon v0.12.16
     Compiling once_cell v1.20.1
     Compiling cfg-if v1.0.0
     Compiling memchr v2.7.4
     Compiling shlex v1.3.0
     Compiling crossbeam-utils v0.8.20
     Compiling cc v1.1.22
     Compiling fnv v1.0.7
     Compiling ident_case v1.0.1
     Compiling strsim v0.11.1
     Compiling num-traits v0.2.19
     Compiling portable-atomic v1.9.0
     Compiling serde v1.0.210
     Compiling smallvec v1.13.2
     Compiling lock_api v0.4.12
     Compiling aho-corasick v1.1.3
     Compiling pkg-config v0.3.31
     Compiling rayon-core v1.12.1
     Compiling either v1.13.0
     Compiling byteorder v1.5.0
     Compiling regex-syntax v0.8.5
     Compiling quote v1.0.37
     Compiling syn v2.0.79
     Compiling pyo3-build-config v0.21.2
     Compiling crossbeam-epoch v0.9.18
     Compiling parking_lot_core v0.9.10
     Compiling crossbeam-deque v0.8.5
     Compiling regex-automata v0.4.8
     Compiling darling_core v0.20.10
     Compiling onig_sys v69.8.1
     Compiling getrandom v0.2.15
     Compiling memoffset v0.9.1
     Compiling matrixmultiply v0.3.9
     Compiling pyo3-ffi v0.21.2
     Compiling scopeguard v1.2.0
     Compiling heck v0.4.1
     Compiling paste v1.0.15
     Compiling zerocopy-derive v0.7.35
     Compiling serde_derive v1.0.210
     Compiling darling_macro v0.20.10
     Compiling darling v0.20.10
     Compiling zerocopy v0.7.35
  error: failed to run custom build command for `pyo3-ffi v0.21.2`
  
  Caused by:
    process didn't exit successfully: `/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmpsc_vzzhj/tokenizers-0.20.1/bindings/python/target/release/build/pyo3-ffi-13575824b7184c25/build-script-build` (exit status: 1)
    --- stdout
    cargo:rerun-if-env-changed=PYO3_CROSS
    cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
    cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
    cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
    cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
    cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
  
    --- stderr
    error: the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12)
    = help: please check if an updated version of PyO3 is available. Current version: 0.21.2
    = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmplqbh9ypf/.venv/bin/python" PYTHON_SYS_EXECUTABLE="/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmplqbh9ypf/.venv/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmpsc_vzzhj/tokenizers-0.20.1/bindings/python/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/tokenizers.tokenizers.cpython-313-darwin.so"`
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/private/var/folders/zk/3mjwt95x64l9hyn7q711m2xc0000gn/T/tmplqbh9ypf/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
  

  at /opt/homebrew/Cellar/poetry/1.8.4/libexec/lib/python3.13/site-packages/poetry/installation/chef.py:164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with tokenizers (0.20.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "tokenizers (==0.20.1)"'.

  - Installing types-beautifulsoup4 (4.12.0.20240907)
  - Installing virtualenv (20.26.6)
(venv) kamaleshseervi@Kamaleshs-MacBook-Pro gptme % 

[ErikBjare]

tokenizer isn't supported in Python 3.13: #198

But it seems to have finally been merged 5 days ago: huggingface/tokenizers#1639 (comment)

Use Python 3.10-3.12

[Kamalesh-Seervi]

yeh fine I will downgrade and work around cool thanks for quick response