Add Vosk offline speech recognition service #186917

sbruder · 2022-08-16T09:10:02Z

Description of changes

This adds Vosk, an offline speech recognition service. It is part of Summer of Nix to allow live transcriptions with Jitsi Meet, for which Vosk is required.

It consists of several parts:

The main vosk library, including alpha cephei’s fork of kaldi, which is incompatible with the current kaldi derivation in nixpkgs
The python bindings/library to vosk (includes vosk-transcriber command line tool)
- There are also other bindings available, but the python ones are required for vosk-server, so I only packaged them
- I’m not sure if this is the correct way to handle a library that also includes (multiple) language bindings. I wanted to separate them and not put the library itself in python-modules, because it provides bindings for a lot of languages.
The language models for vosk. I packaged the main model directly provided by Vosk, when this is not available the small model and/or a third-party one
The vosk-server WebSocket server
A NixOS module for vosk-server
A NixOS integration test for vosk-server
- It does not use any of the pre-packaged models because their memory consumption is too high and updates to them could make the test fail

Things done

sbruder · 2022-08-16T09:30:34Z

I just saw #185255, which also packages vosk(-api), but uses upstream Kaldi (with a minimal amount of patches) instead of using alpha cephei’s fork. Is that approach preferable?

pkgs/top-level/all-packages.nix

SuperSandro2000 · 2022-08-16T13:19:53Z

pkgs/tools/audio/vosk/kaldi.nix

+    homepage = "https://github.com/alphacep/kaldi/tree/vosk";
+    description = "Kaldi fork for the Vosk offline speech recognition API";
+    license = licenses.asl20;
+    platforms = platforms.unix;


meta.maintainer must be set.

Does it also need to be set for derivations that are not called in all-packages.nix?

Yes, so if I do nix repl and try to get the maintainer, I can find you :P

pkgs/tools/audio/vosk/kaldi.nix

SuperSandro2000 · 2022-08-16T13:21:43Z

pkgs/tools/audio/vosk/kaldi.nix

+  # When updating, always update the git revision (rev) and the version base.
+  # The version base can be obtained by running src/base/get_version.sh in the kaldi source.
+  # Only the first three components (major.minor.patch) should be put in here,
+  # the last one (short git commit hash) will automatically be appended
+  version_base = "5.5.1046";
+  rev = "76cd51d44c0a61e3905c35cadb2ec5f54f3e42d1";
+in
+stdenv.mkDerivation rec {
+  pname = "kaldi";
+  version = "${version_base}-${lib.substring 0 5 rev}";


Suggested change

# When updating, always update the git revision (rev) and the version base.

# The version base can be obtained by running src/base/get_version.sh in the kaldi source.

# Only the first three components (major.minor.patch) should be put in here,

# the last one (short git commit hash) will automatically be appended

version_base = "5.5.1046";

rev = "76cd51d44c0a61e3905c35cadb2ec5f54f3e42d1";

in

stdenv.mkDerivation rec {

pname = "kaldi";

version = "${version_base}-${lib.substring 0 5 rev}";

in

stdenv.mkDerivation rec {

pname = "kaldi";

version = "5.5.1046";

That separates the version base from the git revision which makes it more likely that an update forgets to update the version base. Also, I think the comment should stay there because it’s not very intuitive to see what exactly 1046 is and how to get that for different (newer) versions.

We do not do this for versions in nixpkgs. Please refer to the contributing guide for details

SuperSandro2000 · 2022-08-16T13:21:53Z

pkgs/tools/audio/vosk/kaldi.nix

+    ];
+  });
+
+  # pinned in kaldi/tools/Makefile


Is there a good reason to pin this?

That is for upstream to decide. How could this be not pinned in nixpkgs? Every fetcher/FOD needs an output hash.

This is not the same number as in line 14, is it?

pkgs/tools/audio/vosk/kaldi.nix

pkgs/tools/audio/vosk-server/default.nix

pkgs/development/python-modules/vosk/default.nix

SuperSandro2000 · 2022-08-16T13:24:38Z

nixos/modules/services/audio/vosk.nix

+    };
+    interface = mkOption {
+      type = types.str;
+      default = "0.0.0.0";


Suggested change

default = "0.0.0.0";

default = "127.0.0.1";

0.0.0.0 is the upstream default, which is why I copied it. Because there is a firewall active by default on NixOS, there should be no security issues with keeping that default. Also, many other services are listening on all interfaces by default. Is there a reason why this service in particular should only listen on localhost?

This is more of a secure default even if firewall is disabled. Otherwise, if we really want this: if config.networking.firewall.enable then "0.0.0.0" else "127.0.0.1" should be the good option.

We shouldn't do such a confusing conditional.

Therefore, we should have 127.0.0.1 by default IMHO.

cleeyv

I successfully did a test deployment of this module as a transcription service for the Jigasi component of Jitsi Meet! Everything went smoothly on the Vosk side of this deployment so I don't have any changes to suggest here based on that test. The resulting transcription output of a Jitsi Meet conference is kind of buggy, but this is consistent with an existing upstream issue so doesn't seem related to this package/module: alphacep/vosk-server#89

One thing to note for anyone else trying to deploy this is that it uses a lot of memory. I had problem from hitting memory limits on an 8GB VM and had to redeploy on a 16GB one. Looking at performance graphs, it seems to be using max of nearly 10GB for the first ~5mins after the initial deployment (the starting of the vosk service) and then it drops down to a persistent usage of closer to 9GB. If we add higher level documentation of how to use this service with Jigasi and Jitsi Meet then I think it will be worth mentioning this memory usage, but I don't see a requirement for it in this PR.

sbruder · 2022-08-24T13:09:26Z

@cleeyv The high memory usage is definitely something that should be noted, maybe in the description of the model option. As an alternative for people with less memory, we could also additionally package the small models for languages for which Vosk provides high-quality (but large in size and memory consumption) models. I currently only packaged the large models for those languages. The NixOS integration test currently uses a small model for (among others) that reason, because I could not get it to work with less than 8 GiB of memory, which is more than any other currently existing test in nixpkgs uses.

dit7ya · 2022-09-02T17:13:00Z

Looking forward to this! A working offline ASR opens up so many possibilities.

pkgs/development/python-modules/vosk/default.nix

RaitoBezarius

Thank you for the amazing packaging.
I would need you to move the release notes to 23.05 and some minor things left to be addressed. Other than this, I feel like we can merge it.

jtojnar · 2022-12-01T14:35:18Z

pkgs/tools/audio/vosk/kaldi.nix

+  version_base = "5.5.1046";
+  rev = "76cd51d44c0a61e3905c35cadb2ec5f54f3e42d1";
+in
+stdenv.mkDerivation rec {


Alternately, we might want to override the already existing kaldi package (either src or patches) like in https://github.com/NixOS/nixpkgs/pull/185148/files#diff-a4997bb1585454092e3e7a29bb551970a156f4c7fb1135984c17303027f022e5R10-R32

That way we will not have unnecessary duplication of code that will inevitably get out of sync.

nixos-discourse · 2023-03-22T03:05:36Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-03-20-nixpkgs-architecture-team-meeting-33/26547/2

SuperSandro2000 · 2023-03-30T14:33:11Z

nixos/doc/manual/from_md/release-notes/rl-2211.section.xml

+      <listitem>
+        <para>
+          <link xlink:href="https://alphacephei.com/vosk/">vosk</link>,
+          an offline speech recognition service. Available as
+          <link linkend="opt-services.vosk.enable">services.vosk</link>.
+        </para>
+      </listitem>


Please rebase and delete this change because the file got deleted

SuperSandro2000 · 2023-03-30T14:33:48Z

pkgs/tools/audio/vosk/default.nix

+
+  installPhase = ''
+    install -D libvosk${stdenv.hostPlatform.extensions.sharedLibrary} $out/lib/libvosk${stdenv.hostPlatform.extensions.sharedLibrary}
+    '';


Suggested change

'';

'';

sbruder requested review from FRidh and jonringer as code owners August 16, 2022 09:10

ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 1-10 10.rebuild-linux: 11-100 labels Aug 16, 2022

SuperSandro2000 reviewed Aug 16, 2022

View reviewed changes

sbruder force-pushed the vosk branch from b8b816d to 6a91934 Compare August 16, 2022 14:55

ofborg bot added 10.rebuild-linux: 1-10 and removed 10.rebuild-linux: 11-100 labels Aug 16, 2022

cleeyv approved these changes Aug 23, 2022

View reviewed changes

bobby285271 added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Aug 24, 2022

sbruder force-pushed the vosk branch 2 times, most recently from 237c6a2 to fde0714 Compare September 16, 2022 11:47

jtojnar reviewed Sep 16, 2022

View reviewed changes

pkgs/development/python-modules/vosk/default.nix Outdated Show resolved Hide resolved

sbruder force-pushed the vosk branch from fde0714 to f2dc81c Compare September 16, 2022 12:19

sbruder added 4 commits September 16, 2022 15:25

vosk: init at 0.3.43

b2da849

python3Packages.vosk: init

d7612c2

vosk-server: init at unstable-2022-04-01

02895c2

vosk.models: init

aaefc98

sbruder force-pushed the vosk branch from f2dc81c to d028bbb Compare September 16, 2022 13:25

nixos/vosk: init

5c30147

nixos/tests: add vosk test

121c44f

sbruder force-pushed the vosk branch from d028bbb to 121c44f Compare September 16, 2022 13:27

sbruder mentioned this pull request Sep 16, 2022

peertube: add vosk plugin #191497

Open

13 tasks

bobby285271 removed the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Sep 16, 2022

reivilibre mentioned this pull request Sep 28, 2022

nerd-dictation: init at unstable-2022-07-12 #185148

Draft

13 tasks

SuperSandro2000 added the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 3, 2022

RaitoBezarius approved these changes Dec 1, 2022

View reviewed changes

jtojnar reviewed Dec 1, 2022

View reviewed changes

SuperSandro2000 reviewed Mar 30, 2023

View reviewed changes

anpandey mentioned this pull request Jun 16, 2023

Add numen to nixpkgs anpandey/numen-nix#2

Open

wegank added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Sep 7, 2023

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 19, 2024

wegank marked this pull request as draft March 20, 2024 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Vosk offline speech recognition service #186917

Add Vosk offline speech recognition service #186917

sbruder commented Aug 16, 2022

sbruder commented Aug 16, 2022

SuperSandro2000 Aug 16, 2022

sbruder Aug 16, 2022

RaitoBezarius Dec 1, 2022

SuperSandro2000 Aug 16, 2022

sbruder Aug 16, 2022

SuperSandro2000 Mar 30, 2023

SuperSandro2000 Aug 16, 2022

sbruder Aug 16, 2022

SuperSandro2000 Mar 30, 2023

SuperSandro2000 Aug 16, 2022

sbruder Aug 16, 2022

RaitoBezarius Dec 1, 2022

SuperSandro2000 Dec 3, 2022

RaitoBezarius Dec 3, 2022

cleeyv left a comment

sbruder commented Aug 24, 2022

dit7ya commented Sep 2, 2022

RaitoBezarius left a comment

jtojnar Dec 1, 2022

nixos-discourse commented Mar 22, 2023

SuperSandro2000 Mar 30, 2023

SuperSandro2000 Mar 30, 2023

Add Vosk offline speech recognition service #186917

Are you sure you want to change the base?

Add Vosk offline speech recognition service #186917

Conversation

sbruder commented Aug 16, 2022

Description of changes

Things done

sbruder commented Aug 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cleeyv left a comment

Choose a reason for hiding this comment

sbruder commented Aug 24, 2022

dit7ya commented Sep 2, 2022

RaitoBezarius left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nixos-discourse commented Mar 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment