[3DS] 32x crashes with dynarec on #135
Comments
|
I hear the core is crashing with 32x games on Vita, as well, so this issue may affect beyond just 3DS. |
|
Yes, OpenDingux platforms experience the same issue... It might be worth raising this over at irixxxx's fork: https://github.com/irixxxx/picodrive (most of the recent updates to the core come from here) I believe he is only interested in the stand-alone builds, but he seems to be a wizard with this codebase, and he may have some sympathy for RA... |
I suspect that may be a pragmatic choice on their part, as they had been sending us PRs at the same time as upstream, but it was quite a hassle for them. Hopefully this is something simple, though, and they can at least point us in the right direction. |
|
Thanks for the recommendation, I raised this over there. |
|
ARM DRC crashes are due to the cache handling. I accidently changed that to the libc clear_cache function, disabling the cache handling for non-linux systems in libretro.c. Corrected in my repo. @jdgleaver re opendingux: what exactly is happening? Where can I find more info on this?
Ah, I'm not only in standalone builds. In fact I'm doing a more or less regular picodrive libretro build on linux (x86, RPi1 and Odroid) and OSX to check it's still working on x86_64, armv6, armv7 and armv8. My problem is more related to limited resources. Besides only having limited spare time, I'm normally not doing Windows stuff (don't have that at home), and my hardware cache is rather limited so I can only test a very limited set of the libretro targets. |
|
@irixxxx Firstly, I do apologise most sincerely for making a mess with those cherry picks. I had no idea that you were interested in the libretro core, and just saw that we had cherry picked your commits in the past - and so I did the same. I see now that this was a big mistake! It won't happen again. Please don't worry - we can very easily revert anything on our side. Is it just my PR that needs to be undone? Or are there previous cherry picks as well? Just let us know what you need us to do, and we can sort it out. We would be incredibly grateful for a PR that includes a proper compilation of your 1.97 fixes - that would be so much better than our clumsy cherry picks. Regarding OpenDingux - I'm afraid my ability to debug the issue is limited. I don't have a network adaptor for the device, so have no command line access or remote gdb. At present, attempting to load any 32x content just crashes RetroArch back to the 'desktop', with nothing of any value in RA's log. Note that I haven't yet tested your commit from yesterday - I'll try this as soon as I get access to my RG350M. We don't have an official release of the OpenDingux build yet - I only started cleaning it up a few weeks ago, and we're also currently moving over to new build infrastructure (many of our more exotic ports haven't been set up). But if you would like to try anything, this is a very recent build: https://drive.google.com/file/d/1fWEYfNIsf_n7cOWW8M47xTkB29qLvD0I/view. Install the OPK as normal, and just copy the Also, I don't want to confuse matters (so please ignore this for now!), but I noticed just last night that some Sega CD titles randomly freeze with the core on OpenDingux - not a crash (the quick menu can still be opened, and content can be closed), but the image hangs and the audio becomes static noise. Perhaps this is something similar to the 32x hang that you fixed recently? I only tested this very briefly though (it's quite rare, and only Silpheed seems to hang with any regularity), so haven't opened an issue for it yet - I will do so if I can find an easily repeatable test case, and once I have a save state of an affected game. |
|
Just for info - with commit irixxxx@69c2251 on OpenDingux, the core gets a little further but RetroArch still crashes to the 'desktop'. These are the final lines of the log - unfortunately not very useful: |
|
OK, let me think a bit about how to handle the repos. I'm no real git buff, but anyway all is not lost, I think. Here's something to produce debug output. Could you please apply the diff and send me the output? |
|
Regarding mips32, just to make sure: what device are you running this on? The Makefile compiles for mips32r2, which is correct JZ4770 devices and higher, but anything older only has mips32r1. |
|
@irixxxx Many thanks for the diff. I'm running this on an RG350M (we only 'officially' support JZ4770 and higher devices). Unfortunately, no real output is produced - here's the full log, up until the point where it crashes: retroarch__2020_10_28__22_34_12.log (I should note that all non-32x content launches correctly) Regarding the repo - if we need to switch over to a new 'master' branch and delete those cherry picked commits from the history of that branch, I don't think that would be a problem. Whatever turns out to be easiest for you, I'm sure we can do it. |
|
It appears to be crashing, apparently in rcache_create if all output is really visible (it might not since something might be Do you have debugging capabilities on your rg350? It would be nice to use a debugger to see where it crashes, either by using a debugger or by enabling core dumps and use a post-mortem debugger. A break on the exit()-function and a backtrace would also be helpful if there's no real crash. I can do it if you don't have the resources. I can't probably do it before the weekend, though. That reminds me, is there a sanctioned way to inject my self-built picodrive core into retroarch? Currently I'm doing this by starting retroarch via the command line with a heap of options. Regarding the repo, I have a secondary upstream branch for libretro. I'm currently experimenting with git to see if I can resolve the issues by taking over all newer commits from my notaz upstream branch. Unfortunately I can't simply merge, since the divergence point is way in the past and it produces a huge amount of conflicts. Going to see if I can resolve this by cherrypicking, but I'm not really hopeful that it would be possible without scrambling the history. Maybe the best way would really be to unroll some stuff, then cherrypick all newer commits from irixxxx/master and libretro/master back. At least that would allow for a clean PR. |
Unfortunately no - I don't own the requisite network adaptor hardware, so I've been limited to printf-style debugging via the log. This has been fine for my own work on RetroArch (I'm somewhat used to working on platforms with no proper debugging support, and I'm very comfortable with the frontend code anyway), but it's not much use for a large unfamiliar codebase like this one. I apologise for my glaring inadequacy here - I fully understand that this bug report must very unhelpful and annoying for you :(
If you would be willing to run a debugger on this, you would have my sincere gratitude. And please - there is absolutely no hurry at all. Please don't let this interfere with your other work and commitments. This weekend, next weekend, whenever - your time is valuable, and I appreciate any that you can spare.
You're building via In that case, all you have to do is copy your self-built core to RetroArch's Once your core is 'installed' via either of these methods, it will be handled exactly like the regular core from the buildbot.
I'm not sure either how to do this elegantly without editing the history - this is generally frowned upon, but making a new branch from libretro/master and deleting commits should be acceptable (i.e. as long as we don't 'corrupt' the history of libretro/master itself). Then you'd have a clean base, and if we merged your PR on top of that, the new branch could then become our default (have to update our buildbot links, but I guess that's not too much trouble). Honestly, you are the primary developer of Picodrive now - we should be doing all we can to accommodate you :) If we can do anything to help fix the mess on our end, just let us know. |
You need an adaptor? Am I missing something here?
Thank you. I was more and more getting the impression a lot of people are not appreciating donating time to the public in any form. There are those who do, fortunately.
Ah, I somehow missed this "install a core" stuff. Thanks for that, it will make working on rg350 easier, since you can't start anything useful via command line while gmenu2x is running.
I'll try to wrap some suggestion up. There's some stuff for me to learn about git, I reckon. It's no standard use case, apparently not covered in any "good practice" guide I've read until now.
Hmm, I never intended this. However, after more than 200 commits I suspect I should accept it, although there are still larger parts of the code base I've never been into, besides lacking the hardware know-how notaz and others have built up. |
No, it's me who was missing something! I had no idea that you could do this via USB - I thought it needed a network connection. Well, that's egg on my face... Here's the output from gdb when running RetroArch: |
|
Since the libretro-ization is already upstream, would it make sense to do a clean break with the libretro fork and just update the upstream libretro-ization? If we need a downstream fork for experimentation/debugging/whatever, we could always re-fork but do a better job of avoiding future divergence. /off-topic |
|
I can't believe what I'm seeing. It looks like it can't write to a statically allocated structure array. Can I pester you a bit more? If so, please dump me the asm code of the function and the registers after the crash: TIA. I'm really grateful for your help. |
|
This is what I get:
Heh - Not as grateful as I am for your help. We'd get absolutely nowhere without an expert of your calibre to debug this. There are very few people around with your knowledge and skill set :) And thanks also for opening my eyes to USB debugging on the RG350 - I've been using that just now to fix a long-standing frontend bug affecting fast-forward support. @hizzlekizzle That sounds like a very reasonable idea - I'd support that! |
|
Thank you for the disassembly. It indicates that apparently t9 is required to hold the called function's address on entry. Could you please try this: |
|
Many thanks for the diff! It seems to fix that particular segfault, but now it crashes elsewhere: |
|
Sorry, an oversight. This additional patch should do the trick: This issue is an effect of PIC code. I never had this with my opendingux standalone builds. Interestingly enough I haven't seen these problems on x86/arm/arm64 PIC builds, maybe due to those having PC-relative addressing modes mips has not. I'll have a closer look at all supported DRC platforms to make sure this isn't happening elsewhere. Besides, the patches need some more scrutiny, since this t9 business is only needed for ABI calls, but not for drc-internal calls. There's possibly a small potential to micro-optimize this. BTW, for further gdb assembly dumps please use: |
|
Just to muddle my elation a bit: that most probably doesn't explain this "host register locked" business on the 3ds. |
|
@irixxxx Thank you so very much! The latest patch fixes everything - 32x games now run without issue. You've made an old man very happy. Bless you for working on this. Performance is a little choppy on the RG350 (~45-47 FPS for 3D games and Kolibri), but I will add an automatic frameskip option to the core which should make things quite acceptable :)
Ah, I see! Yes, the fact that we compile cores as shared objects seems to create a number of issues/problems on MIPS (for example, our gpSP core has no dynarec on OpenDingux - we do have a MIPS dynarec written in assembly, but it only works with static builds...).
Certainly, and thank you for your continued interest! How should we go about getting these fixes into the core, once you're happy with them? Would you prefer to wait until we've resolved the repo issues? It might be easier for you if we avoid muddying the waters with interim commits until we've decided on the correct approach there (I do like hizzlekizzle's suggestion of switching over fully to your fork, but whatever you think is best).
In theory, it's supposed to be possible to debug 3DS software using Citra, but I'm sure how that works out in practice... It's possible to use gdb on a real 3DS: https://gist.github.com/LiquidFenrir/d110f3e7755ffbe82672eda49ae21af2 - would that be useful? Maybe @justinweiss could help with that...? |
|
just a quick note to whom it may concern: https://techpubs.jurassic.nl/manuals/0630/developer/Cplr_PTG/sgi_html/apa.html |
|
Just for info, I've added frame skipping to the core (#138) and it makes 32x content far more playable on the RG350. One thing I've noticed is that the RetroArch core is significantly slower than standalone, at least on the RG350 - e.g. Star Wars Arcade, Kolibri and The Amazing Spider-Man are a slideshow in RetroArch, but are very smooth in standalone. I have no explanation for this, but most games run fine - so even if we can't reach parity, I think the core remains worthwhile for 32x content :)
Thank you very much for this! I don't think we have any devs at the moment who know assembly and who would be willing to work on this, but if we find someone that document will be invaluable! |
|
Yes, there's a gdb stub built into the custom firmware for 3DS, so it's possible to remote debug. I have a good setup for that, happy to help with lookups or anything else that would be useful. |
|
OK, @justinweiss we continue this on your issue ikn my repo. I've posted you there already. Meanwhile, I've had a look at ABI descriptions and compiler outputs with and without -fpic for armv7, armv8, x86_64, mips32, ppc64le and risc-v64. I sincerely believe those are all ok as of now. i386 seems to be unaffected as well - can't find any useful ABI doc, but the generated code looks inconspicuous. |
|
Thank you for the confirmation. I really appreciate your dedication in following up on this. Picodrive is in safe hands with you at the helm. |
With dynarec on, 32x roms crash on launch on the 3DS. This seems to have broken in two ways in the last few months:
Unfortunately I have not been able to bisect closer than that, because many of the intermediate commits won't build for me without changes. Happy to help with symbols or lookups if necessary.
The crash happens because some code changes
pcto point to the first element ofblink_free, which is 0. I haven't been able to find stack breadcrumbs or break anywhere close to the crash. The best clue I have is that it crashes sometime after hitting this block of code, where the registers look very similar to what they look like when it crashes:Here is the full crash dump:
The text was updated successfully, but these errors were encountered: