Self-hosted Jitsi

[UkolovaOlga]

I have been thinking about moving to self-hosted Jitsi for a while now but never got time to dive deep into it (or to even explore it enough to make a proper decision on whether it’s even worth it). I would appreciate help in making preliminary analysis, establishing ways to move forward and defining the required resources.

What questions need answers:

1. What server parameters are needed to set it up (a quick overview of the functionality they have would be much appreciated).
2. If the functionality needs to be set up step-by-step, what can be the ‘mvp’ stack that we could use and within what period of time it can be set up and running. + what can be added after.
3. What are the most problematic parts of jitsi functionality (i heard that waiting lobby is a pain in the ass).
4. Rough estimates on how long the overall setup can take.

Our needs:

• To hold weekly opened calls
• To have sporadic calls between different team members
• Rooms for private chats
• Recordings (audio, video) + streaming in future (with no default limits of 1h recording time)
  Speaking of recordings, I’m currently using my dropbox for storing/automatic uploading of the videos and it has tons of limits which is not handy. I’m sure jitsi has some alternatives that can be used.
• In the future this jitsi can/will be used as a platform for online conferences/meetups. This is a separate issue that probably needs more investigation and playing around to know how can be set up and used.
• Maybe something I’m forgetting, but that can be figured out after initial analysis.

Existing alternatives

1. https://bigbluebutton.org/ (and Greenlight as UI; Github). Good for educational purposes, not so much of a conference call/meetup/dev call thing. Registration/signing up required with an email.
2. https://www.privus.global/ - not open source, weird privacy-related claims.

[rajarshimaitra]

cc @MaxHillebrand

Wasn't wasabi doing something like this? Can you share some feedback?

[UkolovaOlga]

Afaik Wasabi is using Fulmo's server or common Jitsi facilities for their calls.
And Hangouts (this is thus far the go-for choice for their team).

[supertestnet]

Here's the youtube tutorial I mentioned in the jitsi chat for how to self-host jitsi: https://www.youtube.com/watch?v=8KR0AhDZF2A -- not sure if that's useful

[3cola]

I would like to try to host it with nixos, as it has been already packaged, see https://nixos.org/manual/nixos/stable/index.html#module-services-jitsi-meet

[3cola]

after an initial nixos installation ( https://nixos.org/manual/nixos/stable/index.html#sec-installation ), one can enable the packaged nixos service by adding this bit to their configuration.nix :

{
  services.jitsi-meet = {
    enable = true;
    hostName = "jitsi.example.com"; # change this value with the hostname of the machine
    config = {
      enableWelcomePage = true;
      prejoinPageEnabled = true;
      defaultLang = "en";
    };
    interfaceConfig = {
      SHOW_JITSI_WATERMARK = false;
      SHOW_WATERMARK_FOR_GUESTS = false;
    };
  };
  services.jitsi-videobridge.openFirewall = true;
  networking.firewall.allowedTCPPorts = [ 80 443 ];
  security.acme.email = "[email protected]"; #lets encrypt email account
  security.acme.acceptTerms = true;
}

the settings above work if the host is facing directly the internet.
if using a reverse proxy for ssl termination (and maybe to resolve different subdomains on the same ip), one need to have a configuration closer to this:

{
  services.jitsi-meet = {
    enable = true;
    hostName = "jitsi.example.com";
    config = {
      enableWelcomePage = false;
      prejoinPageEnabled = true;
      defaultLang = "en";
    };
    interfaceConfig = {
      SHOW_JITSI_WATERMARK = false;
      SHOW_WATERMARK_FOR_GUESTS = false;
    };
  };
  services.jitsi-videobridge.openFirewall = true;
  networking.firewall.allowedTCPPorts = [ 80 ];
  services.nginx.virtualHosts.${config.services.jitsi-meet.hostName} = {
    enableACME = false;
    forceSSL = false;
  };
}

references:
https://nixos.org/manual/nixos/stable/index.html#module-services-jitsi-meet
https://search.nixos.org/options?channel=20.09&show=services.jitsi-videobridge.enable&from=0&size=50&sort=relevance&query=jitsi
NixOS/nixpkgs#82920

[3cola]

anyway, not all functionnalities seem to be working in the nixos package (no screen sharing, recording) from what I see.
it could be possible to add this functionnalities to the nixos package.
it is most probably easier to go with the docker-compose solution on ubuntu, as it is the official deployment supported by jitsi.

[dr-orlovsky]

We have a server where we can easily run dockerized jitsy, if there is such

[3cola]

I would say that a dedicated instance could be simpler. And also we should keep in mind that jitsi can use quite a lot of ressources, especially when a lot of people are connecting at the same time. (I remember some fulmo instance crashing when more than 80 people where on the same room...)

[UkolovaOlga]

Can you cover the questions that are posted in the initial description from the experience you have? Like how much resources will be needed etc.

[3cola]

Olga, I will answer the questions above tomorrow, I still need to play with my deployment and to read the docs to have a better understanding of jitsi. For now, most functionalities are working with docker (only the recording is not yet working...).

[3cola]

...and the recording is now working...

[3cola]

after moving the vm behind a reverse proxy, I am loosing the recording functionnality (jibri).
lol
I will try to make a custom docker image for jibri, or to patch the jar as described here: jitsi/jibri#246

succeeded to fix the recording with minimal changes :

• changing all references to the internal domain "meet.jitsi" to my actual external domain in ".env", "docke-compose.yml"
• adding extra_hosts resolution for my external domain to my external public ip in "jibri.yml"

[3cola]

Answering the Questions:

1. meet-jitsi is composed of different components:

• web (web ui, mandatory),
• prosody (xmpp chat server, mandatory),
• jicofo (xmpp mandatory component),
• jvb (videobridge, mandatory)
• etherpad (a shared pad, optionnal)
• jibri (broadcasting, recording, optionnal)
• jigasi (audio gateway for SIP, optionnal)

2. In my opinion, etherpad and jigasi are not needed, jibri is a nice to have (as if the recording is done locally, it should work better) but there are other solutions that allow you to record the call ( cloud solution like https://jibrisondemand.com/ or there should even be some chrome plugin that allow this ?)

3. I didn't play with the waiting loby as I disabled authentication for my tests. The configuration of the authentication does not need to be painful: there is an internal authentication available, you can add a "super user" in one command line on the server.
   I guess it is a different story if you want to integrate it with an ldap, but I don't think that this is needed in an MVP.
   In your situation, there would be few authenticated users, we could script it or do it on demand. If the authentication is activated, only an authenticated user can start a room. If not, the first one in the room get to be the admin.
   For me, the most painful part was to setup the reverse proxy / load balancer (I used haproxy) in front of the server running jitsi. You have to setup properly http2 for the websocket. By doing so (using a reverse proxy), you have also to change some configuration in order to make jibri work properly (as jibri try otherwise to connect in https to a http server and crash...)
   Second biggest pain point is jibri, when setup behind a reverse proxy.

4. I would say that the setup would take a day, depending on the flavor of the installation. We can count another day for extensive testing and fine tuning. (I would like to break my jitsi instance with this : https://github.com/chamerling/jitsi-torture-selenium-grid-platform )

Server sizing (very rough estimates):
Estimations: (depends on the cpu, and the target quality of the video stream... of course !)
[1] 1vCPU 2GB RAM (max 4-5 persons) docker
[2] 4vCPU 8GB (20 persons (video) - 30 persons (audio only)) dedicated server or with docker (need to be validated)
[3] 8CPU (quad-core Intel ® Xeon® E5-1620 v2 @ 3.70GHz) 16 GB (100 persons ??) dedicated Server (bare-metal)
[4] 8CPU 16GB (web, prosody, jicofo, etherpad, jibri) dedicated server + x*4CPU 16GB RAM per videobridge (> 1000 persons) dedicated server + kubernetes cluster
count 0.2 cpu per participant, 512MB (idle) + ~128MB per users) + bandwith (5MBps per 1080p video stream serverside
usage can depend on the browser, audio/video/screensharing...
number of file descriptors can be a limiting factor (if more than 100 persons), but it is easy to tune ulimits.
so basically over 20-30 persons, you need to have more than 100 MBps up and down for your server.
see https://community.jitsi.org/t/recommended-server-specs-for-2020/32041/14

I would need to benchmark my current installation with this https://github.com/chamerling/jitsi-torture-selenium-grid-platform to get more precise estimations.

For your needs:
To hold weekly opened calls -> at least [2]
To have sporadic calls between different team members -> [1] or [2]
Rooms for private chats -> at least [1]
Recordings (audio, video) + streaming in future (with no default limits of 1h recording time) -> any setup with Jibri or external services (you can also directly stream to youtube I think, didn't try)
Speaking of recordings, I’m currently using my dropbox for storing/automatic uploading of the videos and it has tons of limits which is not handy. I’m sure jitsi has some alternatives that can be used. -> with jibri, the recordings sits on the server, but you can write a bash script that do something at the end of the recording. We could imagine to push them on an sftp.
In the future this jitsi can/will be used as a platform for online conferences/meetups. This is a separate issue that probably needs more investigation and playing around to know how can be set up and used. -> at least [3], depending on the number of participants [4]
Maybe something I’m forgetting, but that can be figured out after initial analysis. -> You could even think of deploying it in a kubernetes cluster and scale on demand... <- super nice setup :D #myFavorite #totallyUnnecessary

Existing alternatives:
I did not try anything else.
but it could be worth to take a look at https://galene.org/ or janus.