Sapphire RX580 Nitro+ #4

CalcProgrammer1 · 2020-07-14T18:50:51Z

I got this GPU and am trying to figure out how to use this driver to control and reverse engineer it. I figured it would be a simple matter of copying in the correct subvendor ID:

    {0x1002, 0x67df, 0x1da2, 0xe366, 0, 0, CHIP_POLARIS10},     // RX580 (Sapphire Nitro+)

This causes the driver to be detected, but i2cdetect on the bus returns all --

adam@adam-linux-desktop:~/aura-gpu$ sudo i2cdetect 10
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-10.
I will probe address range 0x03-0x77.
Continue? [Y/n]
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

I noticed there are a bunch of errors in dmesg:

[ 8646.596719] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:206] Pre Transaction: addr = 77, offset = 0, rw = w, count = 1, out = 0
[ 8646.596721] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f4 0
[ 8646.596721] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f4 val 8
[ 8646.596724] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 1862 6db6d800
[ 8646.596725] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 1862 val 6db6d800
[ 8646.596726] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 0 val 12260
[ 8646.596728] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 1 0
[ 8646.596728] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 1 val 0
[ 8646.596730] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 1 0
[ 8646.596731] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 1 val 0
[ 8646.596733] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16fb 0
[ 8646.596734] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fb val 2829
[ 8646.596736] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f7 3e80002
[ 8646.596737] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f7 val 3e80002
[ 8646.596739] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f7 3e80002
[ 8646.596739] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f7 val 3e80002
[ 8646.596741] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f5 0
[ 8646.596742] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f5 val 2
[ 8646.596744] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f9 23100
[ 8646.596744] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f9 val 13100
[ 8646.596747] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f9 13100
[ 8646.596747] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f9 val 13100
[ 8646.596749] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f9 13100
[ 8646.596750] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f9 val 23100
[ 8646.596750] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fa val 8000ee00
[ 8646.596751] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fa val 80010000
[ 8646.596752] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fa val 80020000
[ 8646.596753] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fa val 80030000
[ 8646.596755] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f4 8
[ 8646.596755] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f4 val 9
[ 8646.596857] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f6 2
[ 8646.596959] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f6 610
[ 8646.596961] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f6 610
[ 8646.596963] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f5 0
[ 8646.596963] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f5 val 2
[ 8646.596966] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f4 8
[ 8646.596966] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f4 val 2
[ 8646.596970] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16f4 2
[ 8646.596970] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16f4 val 0
[ 8646.596972] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:140] Read reg 16fb 2829
[ 8646.596973] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:152] Writing reg 16fb val 0
[ 8646.596974] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:216] Post Transaction: status = 2, read = 0
[ 8646.596974] lights aura: [/home/adam/aura-gpu/aura-gpu-hw.c:224] hw_i2c error 2

I haven't experimented in Windows (AMD ADL) yet but I will soon.

The text was updated successfully, but these errors were encountered:

twifty · 2020-07-15T00:39:01Z

This driver is only suitable for ASUS cards. If you have Aida64 or MSI AfterBurner, you can easily and quickly dump the cards I2C (They internally use ADL).

CalcProgrammer1 · 2020-07-15T05:32:03Z

I was under the impression that this project was an i2c driver for AMD GPUs. What is Aura-specific about that?

I got the Sapphire card's RGB controller partially reverse engineered. It uses i2c address 0x55. I was able to detect it with OpenRGB's ADL implementation and dump the registers as I changed them with TriXX. I've written a new OpenRGB controller based on this information and on Windows it is working.

What do I need to capture from Aida64/AfterBurner to get this driver working in Linux?

twifty · 2020-07-15T08:11:09Z

The ASUS cards have an IT8915FN chip on the board, visible via i2c, within which the AURA RGB can be controlled. AFAIK not all AMD cards have this chip. This module creates a driver for the known missing i2c bus and looks for that specific chip. While, on sapphire cards, another similar chip may be on the hidden i2c bus, I have no way to verify. The error you are seeing above is the driver trying to read to a chip that is not present on the bus.

An important note, I cannot simply create a driver for that bus because the amdgpu module controls all memory access locks. This driver is accessing an already loaded memory region, but since amdgpu doesn't seem to use the exact address it "shouldn't" cause any problems (kernel crashes). TBH I wish amd would add support for these missing buses, including the vega/navi cards. It would make life so much easier.

This repo exists only as a debugging aid to get it working on vega cards. It is not intended to be a standalone driver. Using it as such would more than likely cause kernel scheduler errors and crashes.

ADL reports the missing bus as #6. If you can confirm the bus and chip address, and also reveal which registers do what, I can add the support to the main lights module. In that case we should move this conversation over to the https://github.com/twifty/lights repo.

twifty · 2020-07-15T08:35:44Z

You might also want to try the first commit of this module. The master branch was reconfigured to work the cards BIOS (which didn't work), while the first commit used direct memory access.

CalcProgrammer1 · 2020-07-16T02:15:03Z

Does that mean that even though this driver exposes a full i2c interface, it will only ever communicate to one address? I'm not seeing how this driver (which appears to be a generic i2c controller driver for the "unused" i2c port on the AMD GPU) is Aura specific. The IT8915FN would be an I2C slave device just like the 0x55 device on the Sapphire GPU would it not?

Is it possible to patch the amdgpu module to access this bus? I assume the code in here is duplicated from the code that accesses the DDC i2c busses?

I know for sure my Sapphire GPU RGB controller is at 0x55 as I can control it with ADL. I've i2cdetect'ed all AMD i2c ports as well as using this driver and was not able to find an 0x55. I also tried blind writing to 0x55 (which I know would change the color if successful) but the color did not change.

Interestingly, using the first commit of this driver I see a device at 0x20. If I i2cdump it I get this:

adam@Adam-Desktop-2:~/aura-gpu$ sudo i2cdump 8 0x20
No size specified (using byte-data access)
WARNING! This program can confuse your I2C bus, cause data loss and worse!
I will probe file /dev/i2c-8, address 0x20, mode byte
Continue? [Y/n] 
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: XX ff ff ff XX XX XX XX XX XX XX XX XX XX XX XX    X...XXXXXXXXXXXX
10: ff XX XX XX XX XX XX XX XX ff XX XX XX XX XX XX    .XXXXXXXX.XXXXXX
20: ff ff XX XX ff ff ff XX XX XX XX XX XX XX XX XX    ..XX...XXXXXXXXX
30: XX XX XX XX XX XX XX XX ff ff XX XX XX XX XX XX    XXXXXXXX..XXXXXX
40: XX XX XX XX XX XX XX XX XX XX ff XX XX XX XX ff    XXXXXXXXXX.XXXX.
50: XX ff ff ff XX ff XX XX XX XX XX XX XX XX XX XX    X...X.XXXXXXXXXX
60: XX XX XX XX XX XX XX XX ff XX XX XX XX XX XX XX    XXXXXXXX.XXXXXXX
70: XX XX XX XX XX XX XX XX ff ff ff ff ff ff XX XX    XXXXXXXX......XX
80: XX XX XX XX XX XX XX XX ff XX XX ff ff ff XX XX    XXXXXXXX.XX...XX
90: XX XX XX XX XX XX ff XX XX ff ff ff XX XX XX XX    XXXXXX.XX...XXXX
a0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
b0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX    XXXXXXXXXXXXXXXX
d0: ff ff ff ff ff ff ff XX XX XX ff XX XX XX XX XX    .......XXX.XXXXX
e0: XX ff ff ff ff ff ff ff XX XX XX XX XX XX XX XX    X.......XXXXXXXX
f0: XX XX XX ff XX XX XX ff ff ff XX ff XX ff ff XX    XXX.XXX...X.X..X

Would this diagram be correct?

twifty · 2020-07-16T03:44:32Z

Forgive me, you are correct, I'm confusing the repos. This repo does exclude the AURA specific code. However it has been contorted to probe the only the vega cards.

From what I remember, the Polaris cards have two hidden buses. 4 are exposed, one for each of the display outs, 2 are completely excluded. Patching amdgpu to expose them would be a massive undertaking. First, the code base is a convulted mess, it doesn't follow the kernel coding guidlines at all (I'm surprised Linus has even allowed it). Second, the module has hard coded structs for each of the chips. Adding an extra i2c would confuse its DC detection. My plan was to get it working on all chips then contact the amdgpu devs and get it included.

Reconfiguring this repo to work with direct memory access on the Polaris chips should expose one of the two hidden i2c buses. However, It will take me a few days to rework the code. (I'm currently eyes deep in extra "unpaid" work, thank you covid). But, as I mentioned earlier, this repo only exists as a test case. It is not intended to be used in other projects and will likely be deleted in the future.

twifty · 2020-07-16T04:49:57Z

I've just ran an i2cdetect on the latest kernel. Looks like a few changes have been made since I last played with this code. Still the four i2c buses visible on my navi card but now they've also exposed three aux (which run over i2c) buses. I've been waiting for amdgpu to mature on navi cards before I start coding again. Looks like that time has come.

CalcProgrammer1 · 2020-07-16T06:21:04Z

IIRC I had i2c-0 to i2c-7 without your driver. 0 was the chipset (i801), two were non-AMD (something like "dmdc" or similar, really short name, can't remember exactly, so the remaining 5 were AMD. My card has 5 outputs (2 HDMI, 2 DP, 1 DVI) so that would make sense.

twifty · 2020-07-16T06:51:15Z

Also to note, BIOS functions should work with the Polaris chipset, but I have not managed to get i2c access to work on navi. This is probably because the i2c algorithm was changed on vega but the kernels legacy ATOM BIOS has not been updated to reflect that change on newer cards. I have what I believe to be the correct memory addresses for vega/navi but without the exact algorithm it's like trying to understand a foreign language given only the vowels.

When I get an opportunity, I'll swap out my GPU for the 580 and revert this repo to a working state. From there I'll use the code base to probe my navi card. Hopefully in doing so I can also solve for vega cards. From this you'll have at least one commit that should expose the hidden bus. But as noted, it is not intended for public release. It WILL cause problems with the existing amdgpu driver.

All this being said, it's also possible for the chips we're looking for to not even be on an i2c bus and that ADL on windows is doing some magic under the hood to make it look like one. Very unlikely though.

CalcProgrammer1 · 2020-07-16T19:54:45Z

I got something interesting from TriXX (Sapphire's software).

https://gitlab.com/CalcProgrammer1/OpenRGB/-/issues/134#note_380909655

Someone posted a .net DLL pulled from TriXX and I decompiled it. It very clearly has a WriteI2C function and is writing the expected values. It also has this configuration block that indicates port is 7. Device 85 is hex 0x55, which is what I got from "i2cdetect" using the ADL I2C interface.

  public class NitroGlowV1 : UserControl, IComponentConnector
  {
    private ApiBase.I2CParameters i2cParameters = new ApiBase.I2CParameters()
    {
      device = 85,
      port = 7,
      speed = 0,
      extraDelay = 0,
      flags = ApiBase.I2CFlags.SKIP_ADL_I2C
    };

Does port = 7 mean anything to you? I figure that should help us figure out which hidden i2c interface it uses?

CalcProgrammer1 · 2020-07-19T19:21:58Z

https://www.phoronix.com/forums/forum/linux-graphics-x-org-drivers/open-source-amd-linux/1180401-switch-off-sapphire-nitro-5700-xt-rgb-leds-linux?p=1194495#post1194495

There is some discussion on Phoronix with one of the AMD driver developers. He suggested it might be the SMU bus. I noticed that none of the amdgpu busses were even showing. Apparently the display controller busses and GPU busses are different? There are two files:

drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c

and

drivers/gpu/drm/amd/amdgpu_dm/amdgpu_dm.c

which both provide i2c interfaces.

The ones in amdgpu_i2c.c:

"AMDGPU i2c hw bus %s"

"AMDGPU i2c bit bus %s"

The ones in amdgpu_dm.c:

"AMDGPU DM i2c hw bus %d"

"AMDGPU DM aux hw bus %d"

My i2cdetect -l:

i2c-3	i2c       	AMDGPU DM i2c hw bus 0          	I2C adapter
i2c-1	smbus     	SMBus PIIX4 adapter port 2 at 0b00	SMBus adapter
i2c-8	i2c       	AMDGPU DM aux hw bus 0          	I2C adapter
i2c-6	i2c       	AMDGPU DM i2c hw bus 3          	I2C adapter
i2c-4	i2c       	AMDGPU DM i2c hw bus 1          	I2C adapter
i2c-2	smbus     	SMBus PIIX4 adapter port 1 at 0b20	SMBus adapter
i2c-0	smbus     	SMBus PIIX4 adapter port 0 at 0b00	SMBus adapter
i2c-9	i2c       	AMDGPU DM aux hw bus 1          	I2C adapter
i2c-7	i2c       	AMDGPU DM i2c hw bus 4          	I2C adapter
i2c-5	i2c       	AMDGPU DM i2c hw bus 2          	I2C adapter

Only the DM busses are showing. I found a flag amdgpu_hw_i2c that defaults to 0. I just built a kernel with it default to 1. Hopefully the other busses will show now.

CalcProgrammer1 · 2020-07-19T20:54:47Z

Success! Sort of... The i2c-10 bus is the right one. I can detect the 0x55 controller and change mode (but not color for some reason).

i2c-0	smbus     	SMBus PIIX4 adapter port 0 at 0b00	SMBus adapter
i2c-1	smbus     	SMBus PIIX4 adapter port 2 at 0b00	SMBus adapter
i2c-2	smbus     	SMBus PIIX4 adapter port 1 at 0b20	SMBus adapter
i2c-3	i2c       	AMDGPU i2c hw bus 0x90          	I2C adapter
i2c-4	i2c       	AMDGPU i2c hw bus 0x91          	I2C adapter
i2c-5	i2c       	AMDGPU i2c hw bus 0x92          	I2C adapter
i2c-6	i2c       	AMDGPU i2c hw bus 0x93          	I2C adapter
i2c-7	i2c       	AMDGPU i2c hw bus 0x94          	I2C adapter
i2c-8	i2c       	AMDGPU i2c hw bus 0x95          	I2C adapter
i2c-9	i2c       	AMDGPU i2c hw bus 0x96          	I2C adapter
i2c-10	i2c       	AMDGPU i2c hw bus 0x97          	I2C adapter
i2c-11	i2c       	AMDGPU DM i2c hw bus 0          	I2C adapter
i2c-12	i2c       	AMDGPU DM i2c hw bus 1          	I2C adapter
i2c-13	i2c       	AMDGPU DM i2c hw bus 2          	I2C adapter
i2c-14	i2c       	AMDGPU DM i2c hw bus 3          	I2C adapter
i2c-15	i2c       	AMDGPU DM i2c hw bus 4          	I2C adapter
i2c-16	i2c       	AMDGPU DM aux hw bus 0          	I2C adapter
i2c-17	i2c       	AMDGPU DM aux hw bus 1          	I2C adapter

adam@adam-linux-desktop:~/linux$ git diff
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a027a8f7b281..ea1e9b4f1684 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -3168,7 +3168,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
                        goto failed;
                }
                /* init i2c buses */
-               if (!amdgpu_device_has_dc_support(adev))
+               //if (!amdgpu_device_has_dc_support(adev))
                        amdgpu_atombios_i2c_init(adev);
        }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 126e74758a34..ddd9d57bfdbd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -101,7 +101,7 @@ int amdgpu_benchmarking = 0;
 int amdgpu_testing = 0;
 int amdgpu_audio = -1;
 int amdgpu_disp_priority = 0;
-int amdgpu_hw_i2c = 0;
+int amdgpu_hw_i2c = 1;
 int amdgpu_pcie_gen2 = -1;
 int amdgpu_msi = -1;
 char amdgpu_lockup_timeout[AMDGPU_MAX_TIMEOUT_PARAM_LENGTH];

CalcProgrammer1 · 2020-07-19T21:08:24Z

I'm pretty sure these are actually the same busses...there seems to be 8 busses total. It looks like the driver initializes just the display controller busses if the device has a display controller.

The DM aux busses seem to be their own independent thing (don't see the same i2cdetect pattern on any of the 0x9x busses).

0x90
0x91 - AMDGPU DM i2c hw bus 2 (Second HDMI port) (I can dump my TV's EDID here)
0x92
0x93 - AMDGPU DM i2c hw bus 3 (First HDMI port)
0x94
0x95 - AMDGPU DM i2c hw bus 4 (DVI port)
0x96
0x97 - Hidden bus with GPU RGB controller

Interesting to note - If I try to dump EDID from the DM interface, it works. If I try to dump EDID from the matching hw interface, it returns all 00. Maybe there's an issue with the hw driver? Maybe if I instead change the display controller code to forcibly init all busses it would work?

CalcProgrammer1 · 2020-07-19T23:09:23Z

Two different i2c xfer functions:

amdgpu_dm.c

static int amdgpu_dm_i2c_xfer(struct i2c_adapter *i2c_adap,
			      struct i2c_msg *msgs, int num)
{
	struct amdgpu_i2c_adapter *i2c = i2c_get_adapdata(i2c_adap);
	struct ddc_service *ddc_service = i2c->ddc_service;
	struct i2c_command cmd;
	int i;
	int result = -EIO;

	cmd.payloads = kcalloc(num, sizeof(struct i2c_payload), GFP_KERNEL);

	if (!cmd.payloads)
		return result;

	cmd.number_of_payloads = num;
	cmd.engine = I2C_COMMAND_ENGINE_DEFAULT;
	cmd.speed = 100;

	for (i = 0; i < num; i++) {
		cmd.payloads[i].write = !(msgs[i].flags & I2C_M_RD);
		cmd.payloads[i].address = msgs[i].addr;
		cmd.payloads[i].length = msgs[i].len;
		cmd.payloads[i].data = msgs[i].buf;
	}

	if (dc_submit_i2c(
			ddc_service->ctx->dc,
			ddc_service->ddc_pin->hw_info.ddc_channel,
			&cmd))
		result = num;

	kfree(cmd.payloads);
	return result;
}

atombios_i2c.c

int amdgpu_atombios_i2c_xfer(struct i2c_adapter *i2c_adap,
		      struct i2c_msg *msgs, int num)
{
	struct amdgpu_i2c_chan *i2c = i2c_get_adapdata(i2c_adap);
	struct i2c_msg *p;
	int i, remaining, current_count, buffer_offset, max_bytes, ret;
	u8 flags;

	/* check for bus probe */
	p = &msgs[0];
	if ((num == 1) && (p->len == 0)) {
		ret = amdgpu_atombios_i2c_process_i2c_ch(i2c,
						  p->addr, HW_I2C_WRITE,
						  NULL, 0);
		if (ret)
			return ret;
		else
			return num;
	}

	for (i = 0; i < num; i++) {
		p = &msgs[i];
		remaining = p->len;
		buffer_offset = 0;
		/* max_bytes are a limitation of ProcessI2cChannelTransaction not the hw */
		if (p->flags & I2C_M_RD) {
			max_bytes = ATOM_MAX_HW_I2C_READ;
			flags = HW_I2C_READ;
		} else {
			max_bytes = ATOM_MAX_HW_I2C_WRITE;
			flags = HW_I2C_WRITE;
		}
		while (remaining) {
			if (remaining > max_bytes)
				current_count = max_bytes;
			else
				current_count = remaining;
			ret = amdgpu_atombios_i2c_process_i2c_ch(i2c,
							  p->addr, flags,
							  &p->buf[buffer_offset], current_count);
			if (ret)
				return ret;
			remaining -= current_count;
			buffer_offset += current_count;
		}
	}

	return num;
}

twifty · 2020-07-20T03:56:52Z

This is looking great. It looks like the kernel has undergone some major changes since I last looked.

The aux bus operates on top of the i2c and is used mainly to communicate with the monitor. I once thought ADL was using aux due the memory constraints it imposes are the same as for aux, but that lead to dead end. Unless something has changed, the hw flag you used defaulted to on and it controlled whether the driver uses the BIOS vs direct memory access. In my earlier tests changing the flag didn't expose the i2c. It was configured to only look for the four known display ports. From the look of your results that may have changed.

Regarding your change color problem. The aura chip doesn't accept bulk writes. It must be written a single byte at a time. Trying to write multiple will set the first but all remaining are dropped without an error. Might be a similar problem on your chip.

After today, my workload is significantly reduced, so I finally have the time to start playing with the kernel again.

CalcProgrammer1 · 2020-07-20T04:44:20Z

Changing the hw flag (defaults off) to on alone didn't do anything for me either. I only got the 0x90-97 busses to show by commenting out the if (!amdgpu_device_has_dc_support(adev)) line so that the amdgpu i2c busses would init even if the device does have a display controller. It's absolutely a hack as now there are two i2c drivers accessing one piece of hardware, but it got the disabled busses to show.

As for the color change issue, I wasn't using bulk writes. On Windows I reverse engineered the chip using the equivalent of i2cdump on ADL I2C. It uses the read byte scheme (equivalent to i2cget 10 0x55 0xXX where 0xXX is the register to read). When I try to dump anything on the 0x9X interfaces, it always returns 0.

To change the color I use the write byte scheme (i2cset 10 0x55 0xXX 0xYY). For mode control, 0xXX is 0x00. For colors, 0x03 for red, 0x04 for green, and 0x05 for blue. Mode has to be set to 0x04 for the custom color to show.

I also noticed a flicker to blue every time I changed modes. Static blue is mode 0x00, so my guess is that the driver is inserting an errant 0x00 into each write and the card is treating it as a mode write. Even reading the device causes the mode to change, so there is definitely something up with the amdgpu i2c driver.

twifty · 2020-07-20T04:55:57Z

Yeah, I noticed that in the patch above. It's basically created a BIOS driven device for each of the i2c buses registered in the BIOS. I'll modify this repo to do the same. It looks like your chip is on the second hidden bus whereas aura is on the first. I'll confirm that after the recode. Once I get it working via the BIOS it'll expose exactly which algorithm is being used, from there I'll create the memory access driver.

The ATOM code hasn't been updated in a very long time. I dare not leave it in place as I don't know how it will behave on newer cards. Besides, it's easier to maintain direct memory access (too many structs and versions in the ATOM codebase).

twifty · 2020-07-20T06:44:13Z

The master branch is hard coded to use line number 6

aura-gpu/aura-gpu-hw.c

Line 204 in d98da6d

args.ucLineNumber = 6;

try changing it to 7 (zero based 8 buses).

I'm currently reading over the code to see what changes need to be made. I'm planning to query the BIOS and create a driver for each bus. ATOM uses a hard coded algorithm for Polaris and older. Looking at the kernel, later cards use a different set of bitmasks.

CalcProgrammer1 · 2020-10-27T05:56:29Z

I just repeated my earlier changes on 5.10-rc1 and it's working! The 0x97 bus now reads and writes correctly and OpenRGB detects the GPU with full control! I sent an email to Greg KH and Alex Deucher (AMD). Hopefully they can get us something in 5.10.

Darxus · 2020-11-19T01:44:55Z

Is there a bug open somewhere for the kernel support this needs, that I could keep an eye on? I'd love to get control of the rgb on my navi. (PowerColor Red Devil RX 5700 XT.)

twifty · 2020-11-19T09:16:31Z

Guys, I'm sorry for not keeping up with my commitments. 2020 has been the year of hell for me and my family. If you are religious at all, please put out a prayer for my daughter. I will slowly get back into these projects, but it's gonna take me some time to get back up to speed.

twifty · 2020-11-19T09:19:10Z

@CalcProgrammer1 Is this the patch you used and is it working correctly? You mentioned above that you had problems changing the colors.

adam@adam-linux-desktop:~/linux$ git diff
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a027a8f7b281..ea1e9b4f1684 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -3168,7 +3168,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
                        goto failed;
                }
                /* init i2c buses */
-               if (!amdgpu_device_has_dc_support(adev))
+               //if (!amdgpu_device_has_dc_support(adev))
                        amdgpu_atombios_i2c_init(adev);
        }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 126e74758a34..ddd9d57bfdbd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -101,7 +101,7 @@ int amdgpu_benchmarking = 0;
 int amdgpu_testing = 0;
 int amdgpu_audio = -1;
 int amdgpu_disp_priority = 0;
-int amdgpu_hw_i2c = 0;
+int amdgpu_hw_i2c = 1;
 int amdgpu_pcie_gen2 = -1;
 int amdgpu_msi = -1;
 char amdgpu_lockup_timeout[AMDGPU_MAX_TIMEOUT_PARAM_LENGTH];

Darxus · 2020-11-19T14:47:55Z

Guys, I'm sorry for not keeping up with my commitments. 2020 has been the year of hell for me and my family. If you are religious at all, please put out a prayer for my daughter. I will slowly get back into these projects, but it's gonna take me some time to get back up to speed.

No worries, I totally understand life can be difficult. I really appreciate the work that has been done on these things, and I'm just trying to figure out how to help, while having very little familiarity.

aquatix · 2020-11-26T09:12:02Z

@CalcProgrammer1 do you know if there's an upstream bug about this? I'm running 5.10-rc5 and cannot see the relevant devices yet, so I hope this will get resolved soonish :)

twifty · 2020-11-26T09:36:18Z

I've started working on this. It looks like the amdgpu kernel module has been completely restructured since I first made this repo. I was hoping to modify existing code, but it looks like I'm going to have to do a complete rewrite. I can't take the risk that a struct has a different layout or a macro is doing something different. It doesn't help the the amdgpu devs don't follow any of the kernel programming guidelines. But at least the new codebase is structured in a much easier to read fashion.

aquatix · 2020-11-26T09:58:03Z

Too bad you have to redo all the things, but good to hear the amdgpu code looks a lot better nowadays. Keeping my eye on this.

CalcProgrammer1 · 2020-11-26T10:10:20Z

@CalcProgrammer1 Is this the patch you used and is it working correctly? You mentioned above that you had problems changing the colors.

adam@adam-linux-desktop:~/linux$ git diff
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index a027a8f7b281..ea1e9b4f1684 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -3168,7 +3168,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
                        goto failed;
                }
                /* init i2c buses */
-               if (!amdgpu_device_has_dc_support(adev))
+               //if (!amdgpu_device_has_dc_support(adev))
                        amdgpu_atombios_i2c_init(adev);
        }
 
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 126e74758a34..ddd9d57bfdbd 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -101,7 +101,7 @@ int amdgpu_benchmarking = 0;
 int amdgpu_testing = 0;
 int amdgpu_audio = -1;
 int amdgpu_disp_priority = 0;
-int amdgpu_hw_i2c = 0;
+int amdgpu_hw_i2c = 1;
 int amdgpu_pcie_gen2 = -1;
 int amdgpu_msi = -1;
 char amdgpu_lockup_timeout[AMDGPU_MAX_TIMEOUT_PARAM_LENGTH];

That is the patch I applied and with 5.10-rc1, it is working just fine for me. I can control the colors on my GPU (in fact I think I wore out the color flash memory with keyboard visualizer...oh well, rather have smooth PC side effects than the ability to save colors, and that works great).

That said, in the email chain I was in, one of the AMD driver developers recommended using the display interface level rather than the hw i2c level for making this modification. I'm not exactly sure how to do that. It also seems others using the patch I used are having the issue I experienced with 5.8, where the interfaces show but all reads/writes to them are zero.

twifty · 2020-11-26T10:29:54Z

@CalcProgrammer1 Not sure why they would recommend using the display interface rather than i2c. From my previous testing, the display interface communicates with the monitor and not the GPU. Maybe I'm missing something. TBH, I wish they would just expose all interfaces from within the kernel rather than having to patch or make standalone modules.

CalcProgrammer1 · 2020-11-26T10:42:06Z

Each display connector (HDMI, DP, DVI) has an i2c interface tied to it for EDID. The display manager code has its own i2c driver separate from the hw i2c driver, though it seems both are driving the same i2c hardware. The display manager code only brings up the i2c interfaces associated with a display connector in VBIOS while the i2c driver in the hw code brings up all i2c interfaces.

Darxus · 2020-11-26T17:33:02Z

Is it worth tagging in an AMD developer? @agd5f
I guess he's not actually on this site.

twifty · 2020-11-26T17:44:54Z

@Darxus No need at this point in time. I'm slowly stripping out the unneeded code. I'll probably upload under another repo name when I've got it to a somewhat working state.

Darxus · 2020-11-26T18:30:38Z

I am excitedly looking forward to it.

CalcProgrammer1 · 2020-11-26T18:46:07Z

I think it is a good idea to tag in @agd5f. That's Alex, right? He was who I was talking to in email, along with Greg KH.

…

On Thu, Nov 26, 2020 at 12:30 PM Darxus ***@***.***> wrote: I am excitedly looking forward to it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIAY7GTOHD3DXPUW73BLGDSR2NGBANCNFSM4OZZPDIA> .

twifty · 2020-11-26T19:53:02Z

It might be an idea. All pre vega cards are capable of using the atombios, later cards are configured to use atomfirmware. The problem is, I cannot find anything in the firmware related to i2c. While the new code should work for all older cards, I'm doubtful anything later is going to be a success.

Currently, amdgpu creates a DM (Display Manager) aux and i2c for each each interface (3 DP and 1 HDMI on my 5700xt). The kernel is hard coded with memory address for each of the interfaces, but the two "hidden" i2c interfaces are not present (for navi at least).

When I first had success on the RX580, I found the addresses and algorithm by invoking the atom bios functions, then created a DM i2c for the hidden bus. The same just didn't work on navi. I've even tried finding the i2c methods in the Windows driver assembly, those used by AMDADL, but without being able to plug it into a debugger that proved unsuccessful.

My point here is, while I can create a driver, it will only be compatible with pre vega cards and runs the risk of memory collisions with amdgpu. Trying to patch the kernel to expose the hidden i2c with DM would require modifying several files. First the max_links needs to be modified to allow all possible interfaces (8 if I remember correctly) then memory addresses need to be added for each missing link and for each chipset. I really don't know if the devs are open to a more permanent solution. The best course right now is to apply the above patch and force the i2c to use the bios functions.

If I had access to the AMD whitesheets, related to i2c, this would be a breeze.

CalcProgrammer1 · 2020-11-26T20:00:28Z

Yeah, I think we need to get the kernel devs involved. It seemed like they were willing to help, and it also seems like this is something that should be done in amdgpu rather than as a separate add-on driver. Kernel development mostly happens in email though, not github issues. If you want, I can forward you the email threads I've been in @twifty and you can restate what you've just said about newer cards there. The newest AMD GPU I have access to right now is the RX580, but I plan on buying a 6900XT if and when they are available and would like to control its RGB on Linux.

twifty · 2020-11-26T20:08:55Z

@CalcProgrammer1 The 6900XT is also on my wishlist, I really regret buying the 5700XT.

I haven't yet gone through all the kernel code, there have been many changes since I last worked on this. Before getting the devs involved, I'd like to try a few things and at least make a proper kernel patch. If I can get DM to expose the busses for the older cards, it should make things a lot easier for the newer cards. Those with the insider knowledge should be able to simply add the required addresses and possible algorithm.

twifty · 2020-11-27T12:54:40Z

New repo here https://github.com/twifty/atom-gpu-i2c

It is NOT tested! I need to swap out my GPU in order to test, but I will not have time until after the weekend! If anybody is brave enough though...

NyaomiDEV · 2021-01-02T16:34:19Z

Any news on this?

CalcProgrammer1 · 2021-01-02T20:14:06Z

@AryToNeX Just tested...

[71742.766760] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/device.c:483] register mmio base: 0xFCE00000

[71742.766763] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/device.c:484] register mmio size: 262144

[71742.766769] amdgpu 0000:26:00.0: No more image in the PCI ROM
[71742.766785] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/device.c:44] ATOMBIOS detected
[71742.766787] ATOM BIOS: 113-1E3660U-O51
[71742.766790] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/device.c:379] atom firmware requested 007fffe0 32kb

[71742.766792] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/main.c:23] Failed to create the gpu device
[71742.766793] lights amd-gpu-i2c: [/home/adam/atom-gpu-i2c/main.c:40] Failed to find a valid pci device

It didn't find my GPU but I double checked that it's in pci_ids.h. Not sure why this is happening.

Overc1ocker · 2021-02-23T05:33:29Z

@CalcProgrammer1 @twifty Any progress on this? Interested in testing the atom i2c enabler code with a Gigabyte/Aorus RX580 GPU. (They are practically the same thing) I have a feeling I know what i'm looking for. Most of the Gigabyte cards presumably contain the same controller.

paultop6 · 2021-07-09T14:32:00Z

I have a Sapphire 6700xt Nitro+ and would be willing to test what is being discussed here if that helps?

CalcProgrammer1 · 2021-07-09T14:33:39Z

Pretty sure the i2c driver situation is different on Navi cards. Can you do i2cdetect -l and post the output?

…

On Fri, Jul 9, 2021, 9:32 AM Paul Topley ***@***.***> wrote: I have a Sapphire 6700xt Nitro+ and would be willing to test what is being discussed here if that helps? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIAY7HBS7CK47GNCEWUPCDTW4B6VANCNFSM4OZZPDIA> .

paultop6 · 2021-07-09T14:35:33Z

i2c-3   i2c             AMDGPU SMU                              I2C adapter
i2c-10  i2c             AMDGPU DM aux hw bus 2                  I2C adapter
i2c-1   smbus           SMBus PIIX4 adapter port 2 at 0b00      SMBus adapter
i2c-8   i2c             AMDGPU DM aux hw bus 0                  I2C adapter
i2c-6   i2c             AMDGPU DM i2c hw bus 2                  I2C adapter
i2c-4   i2c             AMDGPU DM i2c hw bus 0                  I2C adapter
i2c-2   smbus           SMBus PIIX4 adapter port 1 at 0b20      SMBus adapter
i2c-0   smbus           SMBus PIIX4 adapter port 0 at 0b00      SMBus adapter
i2c-9   i2c             AMDGPU DM aux hw bus 1                  I2C adapter
i2c-7   i2c             AMDGPU DM i2c hw bus 3                  I2C adapter
i2c-5   i2c             AMDGPU DM i2c hw bus 1                  I2C adapter

i2c-dev and i2c-piix4 modules loaded.

paultop6 · 2021-07-09T14:42:42Z

From what I can tell, reading the various other issues etc around this, there is no device registered at the addresses for nitro glow v1 or v3 (0x55 and 0x28 respectively)

CalcProgrammer1 · 2021-07-09T14:45:14Z

Guessing the bus is missing on Navi as well then.

NyaomiDEV · 2021-07-09T22:32:31Z

Sad that this is not solved still, though

CalcProgrammer1 · 2021-07-09T23:55:22Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

NyaomiDEV · 2021-07-11T17:57:12Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Here on Arch the bus shows up but the colors are all over the place, so I ended up not bothering in the end

paultop6 · 2021-07-12T19:20:55Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Has the patch had to change much in line with new kernel versions?

CalcProgrammer1 · 2021-07-12T19:41:08Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Here on Arch the bus shows up but the colors are all over the place, so I ended up not bothering in the end

For some reason my little patch doesn't work for accessing the 0x00 register, which is mode select on the Sapphire Nitro Glow V1 controller. If your card isn't already in the custom color mode it won't do anything when writing the color registers. My solution was to put the card in custom color mode on Windows and it should save that mode so then colors will work on Linux.

paultop6 · 2021-07-12T19:45:03Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Here on Arch the bus shows up but the colors are all over the place, so I ended up not bothering in the end

For some reason my little patch doesn't work for accessing the 0x00 register, which is mode select on the Sapphire Nitro Glow V1 controller. If your card isn't already in the custom color mode it won't do anything when writing the color registers. My solution was to put the card in custom color mode on Windows and it should save that mode so then colors will work on Linux.

Ahh, important nugget there. Are the settings persisted over boot in general? I.e. its not like Asus cards for instances that need to always talk to aura to set argb settings post boot.

CalcProgrammer1 · 2021-07-12T19:50:05Z

The Sapphire Nitro Glow V1 card saves all changes immediately. Colors and mode persist across reboot/power cycle. It doesn't flicker between changes though. In OpenRGB, "direct" mode is only available for devices that do not save the state, as quickly overwriting the saved values wears out the flash. Direct mode is required to use effects software but can be overridden. I used my Keyboard Visualizer app with the Sapphire LED for several days and now it always boots to white because I think the EEPROM/flash cells used for the color values wore out.

…

On Mon, Jul 12, 2021, 2:45 PM Paul Topley ***@***.***> wrote: My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though. Here on Arch the bus shows up but the colors are all over the place, so I ended up not bothering in the end For some reason my little patch doesn't work for accessing the 0x00 register, which is mode select on the Sapphire Nitro Glow V1 controller. If your card isn't already in the custom color mode it won't do anything when writing the color registers. My solution was to put the card in custom color mode on Windows and it should save that mode so then colors will work on Linux. Ahh, important nugget there. Are the settings persisted over boot in general? I.e. its not like Asus cards for instances that need to always talk to aura to set argb settings post boot. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIAY7HNP7D3XXFLDTMIBTDTXNA4VANCNFSM4OZZPDIA> .

paultop6 · 2021-07-12T20:13:59Z

Its important for my use case (well, its a bareable alternative to full control).
I have a MSI motherboard, and by default it has a slow rainbow effect argb sequence from boot. With my sapphire 6700xt (and I believe this is the case for sapphire 5xxx as well) the card can sync its argb from another source via 3 pin header (usually motherboard).

I booted to windows, fired up Trixx software and set it to sync to external source. Thanks to your info about persistence across boot, this setting is now persisted.

Now on my linux machine it syncs to the motherboard. Happy days

ghost · 2021-10-17T02:00:30Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Is there a patch available for RDNA 2 cards? I'm on Arch Linux with an 6600 XT Nitro+ and I just want to be able to turn off the RGB.

paultop6 · 2021-10-17T06:22:46Z

My little hack patch is still working for me on my RX580. I've just been building new kernels with it instead of running Debian's official kernels. I wish it were officially addressed though.

Is there a patch available for RDNA 2 cards? I'm on Arch Linux with an 6600 XT Nitro+ and I just want to be able to turn off the RGB.

For the sapphire 6700xt the settings are persisted across boot. Assuming the 6600xt is the same, you can use trixx software in windows to turn off rgb and then that setting should stay across boot.

mercuriete · 2022-05-25T18:29:59Z

@CalcProgrammer1
I have an sapphire 6900XT Nitro+
Do you know if your kernel patch works on 2022?

So what I understand on this post:
1- patch the kernel
2- run windows to change a setting
3- open openRGB on linux
4- avoid visual map plugin to avoid break flash memory.

Can you confirm this is correct?

offtopic:
this gpu have a ARGB 3 pin connector but my motherboard doesn't have one.
So I tried to make a cable adapter from ARGB to corsair RGB in order to plug it into my corsair node pro
but it didn't work. I am not sure if I need to set that setting on windows first.
right now I am running on ideas, maybe I should try windows.

mercuriete · 2022-05-25T18:58:39Z

nevermind.
I connected the ARGB 3 pin connector to a corsair node pro with a cable.
then I went to windows and on trixx checked "external source"
after that the gpu started to accept commands from my corsair node pro.

thanks for your work and I leave this here if somebody have my same problem.

jonasmalacofilho mentioned this issue Mar 2, 2021

Gigabyte Aorus RX5700XT Graphic card liquidctl/liquidctl#299

Closed

25 tasks

Sapphire RX580 Nitro+ #4

Sapphire RX580 Nitro+ #4

Comments

CalcProgrammer1 commented Jul 14, 2020

twifty commented Jul 15, 2020

CalcProgrammer1 commented Jul 15, 2020

twifty commented Jul 15, 2020 • edited Loading

twifty commented Jul 15, 2020

CalcProgrammer1 commented Jul 16, 2020 • edited Loading

twifty commented Jul 16, 2020

twifty commented Jul 16, 2020 • edited Loading

CalcProgrammer1 commented Jul 16, 2020

twifty commented Jul 16, 2020 • edited Loading

CalcProgrammer1 commented Jul 16, 2020 • edited Loading

CalcProgrammer1 commented Jul 19, 2020

CalcProgrammer1 commented Jul 19, 2020 • edited Loading

CalcProgrammer1 commented Jul 19, 2020 • edited Loading

CalcProgrammer1 commented Jul 19, 2020

twifty commented Jul 20, 2020

CalcProgrammer1 commented Jul 20, 2020 • edited Loading

twifty commented Jul 20, 2020

twifty commented Jul 20, 2020 • edited Loading

CalcProgrammer1 commented Oct 27, 2020

Darxus commented Nov 19, 2020

twifty commented Nov 19, 2020

twifty commented Nov 19, 2020 • edited Loading

Darxus commented Nov 19, 2020 • edited Loading

aquatix commented Nov 26, 2020

twifty commented Nov 26, 2020

aquatix commented Nov 26, 2020

CalcProgrammer1 commented Nov 26, 2020

twifty commented Nov 26, 2020

CalcProgrammer1 commented Nov 26, 2020

Darxus commented Nov 26, 2020 • edited Loading

twifty commented Nov 26, 2020

Darxus commented Nov 26, 2020

CalcProgrammer1 commented Nov 26, 2020 via email

twifty commented Nov 26, 2020

CalcProgrammer1 commented Nov 26, 2020

twifty commented Nov 26, 2020

twifty commented Nov 27, 2020

NyaomiDEV commented Jan 2, 2021

CalcProgrammer1 commented Jan 2, 2021

Overc1ocker commented Feb 23, 2021

paultop6 commented Jul 9, 2021

CalcProgrammer1 commented Jul 9, 2021 via email

paultop6 commented Jul 9, 2021 • edited Loading

paultop6 commented Jul 9, 2021

CalcProgrammer1 commented Jul 9, 2021

NyaomiDEV commented Jul 9, 2021

CalcProgrammer1 commented Jul 9, 2021 • edited Loading

NyaomiDEV commented Jul 11, 2021

paultop6 commented Jul 12, 2021

CalcProgrammer1 commented Jul 12, 2021

paultop6 commented Jul 12, 2021

CalcProgrammer1 commented Jul 12, 2021 via email

paultop6 commented Jul 12, 2021

ghost commented Oct 17, 2021

paultop6 commented Oct 17, 2021

mercuriete commented May 25, 2022 • edited Loading

mercuriete commented May 25, 2022

twifty commented Jul 15, 2020 •

edited

Loading

CalcProgrammer1 commented Jul 16, 2020 •

edited

Loading

twifty commented Jul 16, 2020 •

edited

Loading

twifty commented Jul 16, 2020 •

edited

Loading

CalcProgrammer1 commented Jul 16, 2020 •

edited

Loading

CalcProgrammer1 commented Jul 19, 2020 •

edited

Loading

CalcProgrammer1 commented Jul 19, 2020 •

edited

Loading

CalcProgrammer1 commented Jul 20, 2020 •

edited

Loading

twifty commented Jul 20, 2020 •

edited

Loading

twifty commented Nov 19, 2020 •

edited

Loading

Darxus commented Nov 19, 2020 •

edited

Loading

Darxus commented Nov 26, 2020 •

edited

Loading

paultop6 commented Jul 9, 2021 •

edited

Loading

CalcProgrammer1 commented Jul 9, 2021 •

edited

Loading

mercuriete commented May 25, 2022 •

edited

Loading