Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Memory Map Setup #545

Open
wants to merge 9 commits into
base: develop
Choose a base branch
from
Open

Conversation

JoshuaEagles
Copy link

This PR adds a memory mapping for RDRAM, so that things like the network commands for READ_CORE_MEMORY and WRITE_CORE_MEMORY work for N64 games.

This is a very simple and minimal implementation that only maps RDRAM, if desired this could easily be extended to map the entirety of the N64 addressing space, although I don't know the exact specifics of what memory goes where.

I'm also not certain about the location of the code for this, I have it in main.c so that it get set after the pointer to RDRAM gets initialized, but is there a better location to do this?

@m4xw
Copy link
Collaborator

m4xw commented Jul 23, 2024

whats the use case for this? never heard of those "network commands"

You want to expose it in and with https://github.com/libretro/mupen64plus-libretro-nx/blob/develop/mupen64plus-core/src/device/device.c (needs appropriate bit set tho for the address bus)

@JoshuaEagles
Copy link
Author

Here's the docs for the network commands, https://docs.libretro.com/development/retroarch/network-control-interface/, the two memory reading commands are the only ones that rely on these memory maps.

For my purposes, I'm trying to make it possible to use this core for Multiworld Ocarina of Time randomizers using Archipelago (https://archipelago.gg), the bsnes-mercury core has these memory maps setup and that allows it to used for SNES games in Archipelago. This would also enable other N64 games to work, and purposes other than Archipelago like inventory trackers.

I'll move it to device.c tonight. Thanks for taking a look at this.

@JoshuaEagles
Copy link
Author

JoshuaEagles commented Jul 23, 2024

I moved the memory map setup to device.c, placed it into init_device() which seems like a sane location, runs at almost the same point in execution as how I had it before and that method already deals with memory mappings.

I did look into using the memory mapping struct for setting up the retroarch memory mappings, it seems doable, but I don't currently know how to test regions of the memory other than RDRAM. The RDRAM memory map I've tested pretty carefully with memory offsets used by the Bizhawk script for connecting Ocarina of Time to Archipelago and the memory lines up, accounting for Bizhawk doing memory reads in little endian and applying a XOR 3 to the address before using it (haven't found the reason for that yet).

Not sure what you meant by needing an appropriate bit set for the address bus, looked around and I couldn't find anything that seemed related, sorry, not very familiar with this codebase.

@m4xw
Copy link
Collaborator

m4xw commented Jul 25, 2024

See here


    struct mem_mapping mappings[] = {
        /* clear mappings */
        { 0x00000000, 0xffffffff, M64P_MEM_NOTHING, { NULL, RW(open_bus) } },
        /* memory map */
        { A(MM_RDRAM_DRAM, dram_size-1), M64P_MEM_RDRAM, { &dev->rdram, RW(rdram_dram) } },
        { A(MM_RDRAM_REGS, 0xfffff), M64P_MEM_RDRAMREG, { &dev->rdram, RW(rdram_regs) } },
        { A(MM_RSP_MEM, 0xffff), M64P_MEM_RSPMEM, { &dev->sp, RW(rsp_mem) } },
        { A(MM_RSP_REGS, 0xffff), M64P_MEM_RSPREG, { &dev->sp, RW(rsp_regs) } },
        { A(MM_RSP_REGS2, 0xffff), M64P_MEM_RSP, { &dev->sp, RW(rsp_regs2) } },
        { A(MM_DPC_REGS, 0xffff), M64P_MEM_DP, { &dev->dp, RW(dpc_regs) } },
        { A(MM_DPS_REGS, 0xffff), M64P_MEM_DPS, { &dev->dp, RW(dps_regs) } },
        { A(MM_MI_REGS, 0xffff), M64P_MEM_MI, { &dev->mi, RW(mi_regs) } },
        { A(MM_VI_REGS, 0xffff), M64P_MEM_VI, { &dev->vi, RW(vi_regs) } },
        { A(MM_AI_REGS, 0xffff), M64P_MEM_AI, { &dev->ai, RW(ai_regs) } },
        { A(MM_PI_REGS, 0xffff), M64P_MEM_PI, { &dev->pi, RW(pi_regs) } },
        { A(MM_RI_REGS, 0xffff), M64P_MEM_RI, { &dev->ri, RW(ri_regs) } },
        { A(MM_SI_REGS, 0xffff), M64P_MEM_SI, { &dev->si, RW(si_regs) } },
        { A(MM_DOM2_ADDR1, 0xffffff), M64P_MEM_NOTHING, { NULL, RW(open_bus) } },
        { A(MM_DD_ROM, 0x1ffffff), M64P_MEM_NOTHING, { NULL, RW(open_bus) } },
        { A(MM_DOM2_ADDR2, 0x1ffff), M64P_MEM_FLASHRAMSTAT, { &dev->cart, RW(cart_dom2)  } },
        { A(MM_IS_VIEWER, 0xfff), M64P_MEM_NOTHING, { &dev->is, RW(is_viewer) } },
        { A(MM_CART_ROM, rom_size-1), M64P_MEM_ROM, { &dev->cart.cart_rom, RW(cart_rom) } },
        { A(MM_PIF_MEM, 0xffff), M64P_MEM_PIF, { &dev->pif, RW(pif_mem) } }
    };

f.e. #define MM_RDRAM_DRAM UINT32_C(0x00000000) is the base addr of dram

Cart DOM (the cart image exposed at 0xB0000000) f.e. would be #define UINT32_C(0x10000000) but that addr is in real 0xB0000000 so it needs the uncached memory bit set (0x10000000 | 0xA0000000 = 0xB0000000)

This works for all the mmio regions

@m4xw
Copy link
Collaborator

m4xw commented Jul 25, 2024

(0xA0000000 is also uncached rdram)

@JoshuaEagles
Copy link
Author

Okay, to make sure I'm understanding this correctly (based on some of my own research as well):

  • 0x80000000 - 0x9fffffff is the cached address space
  • 0xA0000000 - 0xBfffffff is the uncached address space
  • The rest of the address space is able to be decided by the game, so it probably doesn't make sense to map any of it for this

Both the cached and uncached address spaces represent the same data and the difference is just caching, so I think it makes sense for them to both be setup as retroarch memory maps so you can read/write either region interchangably.

I have a better idea of how this should look now, I'll get something implemented tomorrow that should have everything in the uncached and cached address spaces mapped to the respective regions.

@JoshuaEagles
Copy link
Author

Made a bunch of changes, I've set it up to construct a struct that defines the mappings to create for Retroarch, and then it uses that struct to create the mappings both starting at 0x80000000 and starting at 0xA0000000. Adding more, if ever needed, would only involve adding a new entry to that struct defining the mapping.

With this iteration I still kept it to only mapping RDRAM and the Cart ROM. Both of these I'm able to test and verify that the pointer is correct and reads and writes data correctly, or in the case of the Cart ROM it has the CONST flag set so writes aren't allowed. The other thing with most of the other data is there being some extra logic around reads and writes for most of it, which there isn't really a good way to represent through the Retroarch memory maps.

@m4xw
Copy link
Collaborator

m4xw commented Jul 30, 2024

 * kuseg   0x00000000 - 0x7fffffff  User virtual mem,  mapped
 * kseg0   0x80000000 - 0x9fffffff  Physical memory, cached, unmapped
 * kseg1   0xa0000000 - 0xbfffffff  Physical memory, uncached, unmapped
 * kseg2   0xc0000000 - 0xffffffff  kernel-virtual,  mapped

for you relevant are kseg0 and kseg1 only. You want to 1. create a mapping of the value of the mapping (kseg0 | addr) and another one for kseg1 | addr
Both are ultimately the same data. Not every addr is valid in kseg0 (cart dom iirc has to be uncached access, so the ROM only works | KSEG1) on hw. assuming i recall that correctly

@m4xw
Copy link
Collaborator

m4xw commented Jul 30, 2024

idaapi.add_segm(0, 0xA3F00000, 0xA3F00028, ".rdreg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4000000, 0xA4001000, ".sp.dmem", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4001000, 0xA4002000, ".sp.imem", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4040000, 0xA4080008, ".spreg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4100000, 0xA4100020, ".dpcreg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4200000, 0xA4200010, ".dpsreg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4300000, 0xA4300010, ".mireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4400000, 0xA4400038, ".vireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4500000, 0xA4500018, ".aireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4600000, 0xA4600034, ".pireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4700000, 0xA4700020, ".rireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA4800000, 0xA480001C, ".sireg", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA5000000, 0xA6000000, ".cartdom2addr1", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA6000000, 0xA8000000, ".cartdom1addr1", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xA8000000, 0xB0000000, ".cartdom2addr2", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xB0000000, 0xB0000000 + ROM_SIZE, ".cartdom1addr2", "DATA", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xBFC00000, 0xBFC007C0, ".pifrom", "CODE", ADDSEG_SPARSE)
idaapi.add_segm(0, 0xBFC007C0, 0xBFC00800, ".pifram", "CODE", ADDSEG_SPARSE)

Thats about the ones you want

@JoshuaEagles
Copy link
Author

Looked around a decent amount and the only resource I could find that talked about which address ranges could be accessed with caching was this page: https://n64brew.dev/wiki/Memory_map, which states that only RDRAM supports it. So I have it setup so only RDRAM is available in KSEG0 and everything else is only available in KSEG1.

I have the entire address space mapped now, I tried to follow the mapping list you gave me, only two things are different from it. In your mapping list the sp mem is split into dmem and imem, but the code here doesn't seem to reflect that and they're both in dev->sp.mem based on the size of the mapping in init_device. The other place it differs is that the list you gave me doesn't have RSP_REGS2. I put it in since I had the mapping information for it, but I can easily remove it if it shouldn't be mapped here.

Also not sure which sections should be read only, a bunch of sections seem to have extra logic that runs when writing to them that wouldn't happen if written through Retroarch. For now I kept it to only ROMs being read only, since those seem like they should be for sure. I can easily mark others as read only as needed.

Did my best to test everything, although I can only verify for sure that the pointers for the N64 rom and DD rom are for sure since I can look at the roms in a hex editor, everything else I just made sure you could read from them correctly at both the start and end of the address ranges, and I double checked the code.

@m4xw
Copy link
Collaborator

m4xw commented Aug 13, 2024

I will check whats to do from there

@JoshuaEagles
Copy link
Author

If there's any changes you want me to make or any other way I can help with this please let me know, although I assume you're just busy, so no worries.

@JoshuaEagles
Copy link
Author

Hi, just checking in, haven't heard back in awhile, any updates on this?

@m4xw
Copy link
Collaborator

m4xw commented Sep 18, 2024

Hi, just checking in, haven't heard back in awhile, any updates on this?

I didnt get to it yet, sorry. Its on my TODO when I come back from vacation start of October unless you have some urgent reason. I figured its basically only you using it for now, so been throwing that down the list.

@m4xw
Copy link
Collaborator

m4xw commented Sep 18, 2024

But just from code quality etc it passes for me, so no worries on that front

@JoshuaEagles
Copy link
Author

Ah, didn't know you were on vacation, sorry. Not a problem though, knowing you still intend to get to this when you have time is enough for me.

It's not quite only me using it, I have the client I'm using this for in a beta state and have some people testing it, but it's definitely still a very small number of users at this point.

@m4xw
Copy link
Collaborator

m4xw commented Oct 24, 2024

This is on the list for when i merge the gliden rebase

@m4xw
Copy link
Collaborator

m4xw commented Nov 7, 2024

Scheduled to go in next, i need a test case tho to try it

@JoshuaEagles
Copy link
Author

Right, I can put something together to make it easy to test, at least for verifying stuff like RDRAM where its easy to read/write and see the correct output. I can get you that after work.

@JoshuaEagles
Copy link
Author

JoshuaEagles commented Nov 8, 2024

Okay, so the way that you test this is by first enabling network commands in retroarch, which required Show Advanced Settings to be enabled (probably already have that) and then scrolling to the bottom of the network section of the settings and enabling it.
image

Then, well I know how to do it on Linux, I'm not sure what OS you use or if you already have a preferred tool for sending data over udp, but you can use netcat to send commands to it. Steps for me are to first open a game on Retroarch (in this case I'm running Ocarina of Time 1.0) with network commands enabled, then running nc -u 127.0.0.1 55355 to connect, and then you can use the READ_CORE_MEMORY and WRITE_CORE_MEMORY commands from there. Excerpt from the Retroarch docs:

READ_CORE_MEMORY <address> <number of bytes>
WRITE_CORE_MEMORY <address> <byte1> <byte2> ...

Then you can poke at values in memory and see stuff happen in the game. There's a sort of swizzle that the memory has when trying to read it so using a memory map like https://wiki.cloudmodding.com/oot/Save_Format isn't perfectly straightforward. Based on that documentation the current health is at 8011A600, but you actually want to run that address through this function first to get the real address to use: (address XOR 3) - (numberOfBytes - 1), which comes out to 8011A602. This is the address that Retroachievements lists as the address for this data, and bizhawk also reverses the swizzle in it's memory interface, so this seems to be expected behavior.

All that to say, if you load a file on a 1.0 OOT rom and Read/Write to 8011A602 you can get/set Link's health, setting it to 0 will kill him. I've tested this RDRAM behavior pretty extensively with my client I wrote (https://github.com/JoshuaEagles/OOT-RetroArch-Archipelago-Client) so I'm pretty confident about it. I can get you addresses to stuff in other games or something if you'd prefer, RetroAchievements has code notes available for a lot of games that are a good source for that.

Example of commands: sending READ_CORE_MEMORY 8011A602 2 returns with READ_CORE_MEMORY 8011a602 30 00 when you have 3 hearts of health, and sending WRITE_CORE_MEMORY 8011a602 00 00 kills Link.

I don't really know how to test most of the other parts, I was able to verify that the start of the ROM data matched what I expected when looking at the ROM in a hex editor, and I did something similar with the DD rom as well.

@m4xw
Copy link
Collaborator

m4xw commented Nov 8, 2024

Okay so all pretty basic.

I just noticed that this will run into issues with the recompiler cache if this is used to modify game code (rather than .data section)

Is there a way we can have a callback or trigger? Might need to extend RA for that, i can take a look

@m4xw
Copy link
Collaborator

m4xw commented Nov 8, 2024

basically we need to cache flush after the frontend did the write

@JoshuaEagles
Copy link
Author

Hey just to confirm, is that a change you want me to make? I'd need a bit more direction if so. If you're still looking into it not a problem, just checking.

@m4xw
Copy link
Collaborator

m4xw commented Nov 26, 2024

Sorry i was sick for a whole week and then i had another project... Feeling bad for the current delays. I dont need you to do anything right now. I just think this API was never designed for any kind of write operation and Ideally i talk to one of the people that wrote the integration in RA regarding this. I assume u want the write functionality in particular?

@m4xw
Copy link
Collaborator

m4xw commented Nov 26, 2024

If you dont need guarantee that the write memory works safely in all circumstances then its more or less okay as-is

@JoshuaEagles
Copy link
Author

All of the writes I need are to data rather than game code. I'm pretty sure game code modifications are typically rom patches rather than being dynamic like this, especially since the ROM is marked as read only so it'd have to be code in RAM, timing that sounds hard.

If you'd prefer to tie up loose ends like this before merging then please do, I don't want to rush this. But, unless I'm misunderstanding something I don't think the recompiler cache will be an issue for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants