Central Issue for porting to windows

[NoamDev]

This a central issue for discussions about porting Mycroft to windows.
cc @el-tocino

[NoamDev]

• fann2 installation is very difficult to install on windows, I think we should fork fann2, build wheels and release them. This has been done very successfully in https://www.lfd.uci.edu/~gohlke/pythonlibs/#fann2.

• PyAudio can be installed with python 3.6.

• Paths in mycroft/configuration/mycroft.conf are not windows compatible. Should we create another configuration file for windows?

[NoamDev]

I installed the fann2 from the link above (actually though pipwin).
Then tried to move on, I changed the paths in the configuration, replace '~' with %USERPROFILE% and opt/ with %APPDATA%.
After that I ran python -m mycroft.messagebus.service.
I had to replace '/tmp' somewhere with a cross-platform equivalent.
Now I found that SIGKILL is unix specific and in windows it should be done a little differently, but since process killing doesn't sound important as long as everything works as expected, I removed it for now.
message bus seem to work.

[forslund]

Nice!

Shouldn't ~ work?

>>> from os.path import expanduser
>>> expanduser('~/hello')
'C:\\Users\\afxxu3/hello'
>>>

[NoamDev]

Oh cool! I wasn't aware of that.

[NoamDev]

windows does not like to connect to 0.0.0.0 (though listening is not a problem)
so the config should be changed here to 127.0.0.1

[NoamDev]

Oh, a problem, maybe.
python didn't understand %appdata% and %userprofile% and instead created directories with those names....
I can change the second to ~, for now, I'll change the first to something harmless.

[forslund]

Haven't tried this but maybe the code should be updated to use os.path.expandvars as well as expanduser. at some point.

[NoamDev]

Is there a way to disable some skills?
Deleting them just cause them to be re-dowbloaded.

[forslund]

You can add them to the list of blacklisted_skills

Edit: they will still be downloaded but won't be started.

[NoamDev]

Thanks, that's sufficient.
I had to also add os.path.normpath in skill loader, but it works.

[NoamDev]

Do you know of windows builds of mimic? Some binary releases?
I have a ubuntu dual-boot, so I'm technically able to cross-compile, but I'd prefer to save the time...

[NoamDev]

Btw, I have skills service up and runing now:)

[forslund]

No mimic windows exe pre-built I'm afraid. You can skip local mimic for the time being and use the Mimic2 (American Male selection on home.mycroft.ai)

[NoamDev]

But I can't pair it since the code should be spoken...
Is there another way of getting a pairing code?

[forslund]

It will be written in the log output. Look for Pairing code: <CODE>

[NoamDev]

Wait, I'll try anyway.
Maybe ot writes it too.

[NoamDev]

After some workarounds with curses, I have text interface.
However, I can't pair, right after function handle_pairing is finished the pairing-skill is being removed..

2020-07-01 18:18:31.521 | INFO     |  7856 | mycroft.skills.padatious_service:train:102 | Training complete.
2020-07-01 18:18:31.528 | INFO     |  7856 | mycroft.skills.padatious_service:train:106 | Mycroft is all loaded and ready to roll!
2020-07-01 18:18:31.733 | INFO     |  7856 | mycroft.skills.skill_manager:_unload_removed_skills:322 | removing mycroft-pairing.mycroftai
2020-07-01 18:18:31.770 | INFO     |  7856 | mycroft.skills.skill_loader:_execute_instance_shutdown:147 | Skill mycroft-pairing.mycroftai shut down successfully
2020-07-01 18:18:31.773 | INFO     |  7856 | mycroft.skills.skill_manager:_unload_removed_skills:322 | removing mycroft-volume.mycroftai

So the pairing code though retrieved, is never spoken, and even if print it, there's no one left to listen, as the skill is being removed.

[forslund]

There's probably some issue in the code checking if a skill has been deleted. Check this method, if something is too isn't adding up: https://github.com/MycroftAI/mycroft-core/blob/dev/mycroft/skills/skill_manager.py#L313

[NoamDev]

Well, it somehow got solved. 🤷‍♂️
But that doesn't mean I have audio of course...
I find that many packages are missing. Probably a tradeoff of not runing an installation script.

[ChanceNCounter]

I applaud the effort and apparent progress, but I figure I should nevertheless paste the counterargument from chat:

it's dependency hell for days. the nature of python on windows means that many stories end in, "just isn't gonna work cross-platform like that."

WSL is a promising solution, because it provides the necessary compatibility layer between headless *nixware and Windows itself. however, it's still at a stage where they really don't want you to dick with windows from within WSL, apart from launching programs, and i've discovered several times that you can trigger opaque, generic BSODs when you try.

i'm convinced at this point that the best way forward for a windows port is to wait until system control via WSL is more thoroughly supported, and less prone to crashes. if you don't try to control the system, mycroft already runs almost-natively under WSL, if you install pulseaudio on windows.

it may be possible to create a windows-native message bus listener, run mycroft itself under WSL, and write a whole separate windows client that responds to the message bus. i haven't pursued that because it's such a clunky solution. be great for you and for me, but for consumers? yeesh.

On the other hand, I am very optimistic about Mycroft under WSL, assuming WSL catches up. Right now, everything works out of the box with WSL+PulseAudio. You just can't do any system control tasks, launching programs, stuff like that. Just caching directory contents in a skill has caused bluescreens. But, if all you wanna do is interact with Mycroft, Q&A skills, control IoT devices, things like that, WSL will do fine.

[ChanceNCounter]

Oh, and, naturally, no X = no GUI =P but Jarbas' TUI works well.

[NoamDev]

@ChanceNCounter
It's not gonna be easy, but I believe it has to be done that way.

[NoamDev]

I don't understand why if audio service loads services like vlc, it uses the audio utils for tts instead..
When does it use the audio services?

[forslund]

It's made to be a pluggable system for different audio libraries to perform audio playback. This is used for example by skills playing news or a playlist of media.

These are loaded as they are available and none is a hard requirement. the "simple" uses the basic playback commandlines defined in the configuration (starting on L53). And can be overridden in any of the system or user configurations.

VLC will be used if it is installed since it offers better handling than the "simple" systems, etc.

These services aren't needed for speech or anything I wouldn't focus on them in the beginning. Finding replacement commandline applications to use to render speech (play_wav and play_mp3 in the configuration)

[NoamDev]

Ok, for now I use ffplay. It's nice:)
Do you think it's time to fork and start coding?

[forslund]

Haven't you already? ;)

[NoamDev]

Is there anything holy with the fann2 version? Why not use 1.1.2?

[NoamDev]

Ok, I started to make it more organized, beginning with requirements installation script.
I created this pr to allow others to participate.

[forslund]

@NoamDev Sorry didn't quite get around to it yet. I've had limited time in front of the computer these past weeks.

[NoamDev]

@forslund That's ok, if you need help with cross-compiling tell me:)

[forslund]

@NoamDev I've built a static binary of mimic for windows and added to the latest mimic-release: Direct link: https://github.com/MycroftAI/mimic1/releases/download/1.3.0.1/mimic_windows_amd64.zip

Works for me(TM) but let me know if there are issues with using it.

[NoamDev]

How can I force it to use mimic1 specifically without unpairing/disconnecting network?

[NoamDev]

@MatthewScholefield
I saw that you're the one who tagged the precise releases.
How do you build them?

[MatthewScholefield]

@NoamDev I run ./build.sh in the repo.

[krisgesling]

How can I force it to use mimic1 specifically without unpairing/disconnecting network?

Select British Male as the voice, this is Mimic 1.
AKA "ap" in the config which stands for "Alan Pope"

[NoamDev]

I run ./build.sh in the repo.

Thanks!
I see it uses pyinstaller which does not support cross-compiling.
So I think the first step towards windows build is a setup file for windows.
Then, we can use pyinstaller the same way.
That means the build process should be done on a windows machine.
Do you think the setup file should be batch or powershell? powershell is much more convenient, but I'm afraid wine does not support powershell out of the box.
Or maybe use py2exe?
Edit: nevermind, pyinstaller does support cross-compilation (it didn't in the past). no need for py2exe which somewhat dead.
I'll try to see if I can compile it to a windows executable.
Edit2: support was removed in 1.5 again but it's possible to be done with wine.

[NoamDev]

@forslund
your mimic executable works very well!
i'd change my win-setup so that it'll download and unpack this.
Edit: done that.

[NoamDev]

Because of multiple edits, I'll clarify about precise:
to compile precise we use pyinstaller
in order to compile to windows one should either:

1. use a windows machine

2. use wine

wine does not support powershell out of the box meaning if we use wine we have to use batch scripts.
What's wrong with batch? In gentle words I don't like batch, it's ugly.

So, the question is: Are you OK with building precise for windows on a windows machine?
please say yes 😅

[ChanceNCounter]

I think there are two ways of looking at that. On the one hand, it makes sense to build on the target platform. On the other hand, it makes sense to ensure that all builds can be performed by the same machines, and I can't see too many people migrating from Linux to Windows =P

IMO, it comes down to how committed you are to maintaining the Windows build. If there's a reasonable chance that it might become abandonware, I think its best shot at attracting a new maintainer would be cross-compilation, so suffer through the batfile. If you expect you'll be doing it for the foreseeable future, do it however's easiest for you. I wouldn't wish a batfile on my worst enemy.

The biggest danger in packaging a FOSS project for an unsupported platform is not that you'll do it wrong. It's that you'll do it wrong and nobody else knows how to fix it. That ends with really irate users looking for help where none exists 😛 you haven't lived until you've seen somebody throw a fit at a dev team over an unofficial package they found in the Arch User Repository!

So I'd base the decision entirely on whether you think you'll still be maintaining the build in a year.

[NoamDev]

@ChanceNCounter Thanks!
I'm indeed not planning to maintain that forever, so I'll work a little more to make it cross-compilable (have I just made the word up?)
Batch, here we come!

[NoamDev]

Hmm, I can use python sometimes, instead. This would both more easy and maintainable.

[NoamDev]

I created windows precise build script and have built windows precise tars 🥳
I still have to make sure it actuallt works, but hay, we are one step ahead!

[NoamDev]

How can I run the tests on precise executable?

[MatthewScholefield]

pip3 install pytest
pytest

[NoamDev]

inside or outside the venv?

[MatthewScholefield]

Inside

[NoamDev]

As you can see in the issue, releasing a new pyache version is required. The latest release is not windows compatible (though the source code is already).

[forslund]

I'm away camping for a couple of days, in case Kris hasn't verified and prepped the release I can look at the pyache release this weekend and do a release (Kris permitting)

[NoamDev]

I'm away camping for a couple of days, in case Kris hasn't verified and prepped the release I can look at the pyache release this weekend and do a release (Kris permitting)

@forslund sure, enjoy yourself!

[NoamDev]

Amm, what's up with that?
@krisgesling already opened a PR, but a review is needed.

[NoamDev]

I feel confused. why precise has so many places to list its requirements?
requirements.txt, requirements/prod.txt and setup.py has its own independent list of requirements..
setup.py doesn't actually use the above txt files if I understand correctly, it doesn't use find_packages or something like that..
and so, build.sh in its turn doesn't use them.
Am I missing anything?

[forslund]

Seems like some merge has gone wrong there, the requirements.txt shouldn't be needed (replaced by the requirements/prod.txt) (and the setup.py should be changed to reference that one)

As far as I know the short list in the setup.py is the list of packages actually needed for building the executable.

Maybe my gist for building the bundle on Arm64 Linux could be of use, though windows is likely quite different so probably not.

Something that's good to know is that Python 3.8 and precise doesn't work.

[NoamDev]

I need to make sure astor's data files get included, astor (a requirement of tensorflow) now uses a file named VERSION.
Ideally, I would add to the spec file:

a.datas += collect_data_files('astor', subdir=None, include_py_files=False)

But the spec tamplate is used for building many scripts and I worry that not all of them use tensorflow..
So is there a reason to worry? If so, is it possible to include those data files only when needed?

[NoamDev]

Hey all, took me a while but I adapted precise for windows support :)
MycroftAI/mycroft-precise#180

[forslund]

Closing Issue since we're archiving the repo