Distributed clocks / SYNC message

[thetooth]

I'm currently working on a robotics project and have been entertaining the idea of replacing CODESYS which I have been using for the RT aspects of the system with an open source stack. Found this library and it's very cool, took maybe all of an hour to adapt the example and get my drive booted and PDO mapped. The problem I now face is its a Delta B3 drive and apparently wont move unless it's receiving some form of sync message.

Delta servo drives only support synchronous operation which is the controller periodically sends the SYNC object (COB-ID = 0x80).

The drive is fully powered up (locked shaft), receiving control word, target position and mode of operation, it's sending back all the right status word bits as well as actual position + velocity. But it refuses to budge. The only configuration difference between this and CODESYS I can see is that its not receiving a distributed clock.

Now the problem is that I only know a little bit about CoE, and next to nothing about EtherCAT itself. Reading what's out there it seems as though SYNC messages and distributed clocks are two different things, if that's true is there a quick and dirty way to generate that payload and get this project moving forward? I assume the only reason we don't already have DC in this library is because it's a non trivial task and unfortunately I'm probably not in the position to contribute one even though I would like to.

Any help would be much appreciated.

[leducp]

DC sync is indeed not trivial, but it is not that complicated too. The main reason of why it is not implemented yet is the lack of setup to test it thoroughly which is a really important aspect for us as the stack is used in industrial and critical software.

About your problem: SYNC is a CAN concept that may be mapped to DC sync when using CoE profile (CAN over EtherCAT) but as my understanding of this part is scarce, it may also be mapped to something else.

Can you send me the manual of your drive? Also can you do a network trace (wireshark or tcpdump) with CODESYS and with KickCAT from start to functional state (or blocked state with KickCAT)? With this I'm pretty sure that we will be able to at least workaround this situation :)

[thetooth]

Thanks, you can find what you're after here: https://thetooth.name/tmp/

Ignore the FRMW commands in the kickcat dump, that's just wishful thinking on my part.

[leducp]

It seems that your module documentation have an update:
https://filecenter.deltaww.com/Products/download/06/060201/Manual/DELTA_IA-ASD_ASDA-B3_UM_EN_20230428.pdf

It says that EtherCAT mode can be either free run or DC so it should work without DC enabled (12.2.2.1 )
Can you confirm that the doc is just an update and is applicable to your case?

If yes: how is the state of the DS402 Can Open state machine? In CoE (CAN over EtherCAT), one have to go to OPERATIONAL (EtherCAT) then CANOpen ON_STATE to be able to move a motor.

In our Ingenia or Elmo example, there is CANOpenStateMachine that is responsible to move set the motor FSM in the right state to control it.

[thetooth]

Yeah I've based my code on that example, here's the output, control word on the left and status on the right in those bin outputs. It looks to be correct to me. Also although the target position is being shown in the tuning software for the servo, the target position on the display of the drive itself is updated once when the EtherCAT link is established and never again.

DEBUG: Frame.cc:221 read() failed
DEBUG: Link.cc:131 Fail to read nominal interface 
DEBUG: Bus.cc:48 5 slave detected on the network
DEBUG: Slave.cc:160 DC!
DEBUG: Slave.cc:160 DC!
TXPDO
RXPDO
DEBUG: Bus.cc:633 SM[0] type 4 - start address 0x1000 - length 1 - flags: 0x20
DEBUG: Bus.cc:643 slave 03ea - size 1 - ladd 0x0000 - paddr 0x1000
DEBUG: Bus.cc:633 SM[1] type 3 - start address 0x0f01 - length 1 - flags: 0x44
DEBUG: Bus.cc:643 slave 03eb - size 1 - ladd 0x0001 - paddr 0x0f01
DEBUG: Bus.cc:633 SM[3] type 4 - start address 0x1480 - length 10 - flags: 0x20
DEBUG: Bus.cc:643 slave 03ec - size 10 - ladd 0x0002 - paddr 0x1480
DEBUG: Bus.cc:633 SM[2] type 3 - start address 0x1180 - length 12 - flags: 0x24
DEBUG: Bus.cc:643 slave 03ec - size 12 - ladd 0x0002 - paddr 0x1180
DEBUG: Bus.cc:633 SM[3] type 4 - start address 0x1480 - length 6 - flags: 0x20
DEBUG: Bus.cc:643 slave 03ed - size 6 - ladd 0x000e - paddr 0x1480
DEBUG: Bus.cc:633 SM[2] type 3 - start address 0x1180 - length 6 - flags: 0x24
DEBUG: Bus.cc:643 slave 03ed - size 6 - ladd 0x000e - paddr 0x1180
 0b0000000000000000 0b0000011000011000
DEBUG: Bus.cc:894 Invalid working counter for slave 1005
ingenia_control.cc:135: something bad happened at 0 delta: 0
 0b0000000010000000 0b0000011000011000
 0b0000000000000110 0b0000011000011000
 0b0000000000000110 0b0000011000011000
 0b0000000000000110 0b0000011001010000
 0b0000000000000110 0b0000011000110001
 0b0000000000000110 0b0000011000110001
 0b0000000000000110 0b0000011000110001
Target: 1988 Actual Position: 1888 Actual Velocity: 0
 0b0000000000001111 0b0000011000110001
Target: 1990 Actual Position: 1890 Actual Velocity: 0
 0b0000000000001111 0b0000011000110001
Target: 1988 Actual Position: 1888 Actual Velocity: 0
 0b0000000000001111 0b0000011000110011
Target: 1992 Actual Position: 1892 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1988 Actual Position: 1888 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1986 Actual Position: 1886 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1986 Actual Position: 1886 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1970 Actual Position: 1870 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1954 Actual Position: 1854 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1974 Actual Position: 1874 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1952 Actual Position: 1852 Actual Velocity: 0
 0b0000000000001111 0b0000001000110111
Target: 1988 Actual Position: 1888 Actual Velocity: 0```

[Rdk-T]

Hi, in the EEPROM of the slave there are configurations about enabling or not the DC sync
Following your traces it seems it's here.

Can you use the tool, eeprom from the repo, to dump the eeprom so that we can check the configurations of the slave ?

[thetooth]

@Rdk-T I think you're on to something, dump is here: https://thetooth.name/tmp/eeprom.bin
If I startup CODESYS and set the drive to free run I get the exact same behaviour.

[Rdk-T]

Alright, we'll look into this next week.

[leducp]

@thetooth I can not download the eeprom.bin file: I got either a 0 bytes file or a 430 forbidden error (with wget)?

[thetooth]

@leducp it should work now, forgot I was running everything as root to avoid setting caps. I have asked Delta about this issue and yet to get a response.

[leducp]

OK I dig a bit in the eeprom and it says that:
" Input DATA is sampled at -> 00: Start of Frame"
"Output DATA is updated at -> 10: DC SYNC0 event"
source EEPROM word 1 - PDI configuration

This configuration should be mirrored iin the register 0x0150 (you can read it with a FPRD from the master if you want).

My understanding is that the slave device requires a DC configuration on the network. We could either add it to the stack or update the eeprom to set the slave config to free run (writing 0 at eeprom word 1).

Do you have an eeprom configuration tool to try?

Forget about this, I mess up when reading the datasheet because there are multiples tabs depending on the slave structure

[thetooth]

I'm up for anything at this point. Btw I am able to move the motor in profile velocity mode just fine, profile position mode is also broken and neither of those are really useful for the application I had in mind.

[leducp]

OK I may have something to try:
I read the CODESYS network trace and it seems that it is not doing something fancy: it send periodically a refresh command for DC sync and let the slaves handle it. There is not specific configuration and KickCAT is pretty much doing the same for this part.

Here is the perdioc frame config:
ARMW , adp 0xfffd (third slave on the network -> is this your slave? the last one ?), ado 0x910, size 8, payload 0x0000000000000000
My guess is that your try with FRMW was good but the targeted slave may have been wrong.

You can try to add something like that in your loop:

uint8_t sync_time[8];
memset(sync_time, 0, 8);
auto process = [](DatagramHeader const*, uint8_t const* state, uint16_t wkc)
{
    if (wkc != bus.slave.count())
    {
        printf("Oops: %d\n", wkc);
    }
    return DatagramState::OK;
};
auto error = [](DatagramState const&){};
link->addDatagram(Command::ARMW, createAddress(0xfffd, reg::DC_SYSTEM_TIME), sync_time, 8, process, error);

[thetooth]

Thanks, there are presently 5 slaves, EK1100 with two cards, followed by two drives. Is the address returned by bus.slaves().at(a1Drive).address correct for this? it seems to be setting outputs on the EL2889 card so off by 1...

[leducp]

Axxx commands (like ARMW) do not work with fixed addresses but with network position: each salve on the network increment the address field by one and when it reached 0 the slave is addressed.
FPxx commands do work with slave addresses set by the master durung the init phase.

I think you can just try the AMRW commands as codesys do it and if it unlock the situation you can mess up / play with the others commands and addressing scheme (especially because we are in the dark and I don't know if something is missing yet) :)

[leducp]

Note that the addressed slave for DC may not be the motor you want to control but the slave that own the network clock (that is used to synchronize the whole system). DC reference clock shall be the first slave on the network (or at least before others slaves that need it).

[thetooth]

Doesn't seem to care, in packet capture I see the DC packet go out with zero time and the master replies with an offset, so it's working properly in one direction.

I also found this in the manual:
"Object 1006h: Communication cycle period
This object is to set the communication cycle, which is the interval between two SYNCs. If you are not using SYNC, set this object to 0."

Seems like an easy fix right? Only I get the following when trying to set that to zero:
DEBUG: Mailbox.cc:241 Abort requested for 1006:0 ! code 06020000 - The object does not exist in the object dictionnary

[leducp]

An easy fix indeeed, but it seems that your device do not support it, maybe because this is a pure CANOpen feature and this one is a CAN over EtherCAT that use the EtherCAT feature for this feature.

I'm digging into the documentation of the DC feature to check if something else is needed. P.I. the doc is freely available:
https://download.beckhoff.com/download/document/io/ethercat-development-products/ethercat_esc_datasheet_sec1_technology_2i2.pdf

Chapter 9 for DC.

[leducp]

OK found something I missed on first analysis:
1/ the register 0x9A0 is configured (this one is responsible for the SYNC0 generation):
DC CycTime0 (0x9a0): 0x001e8480
2/ the register 0x990 is configured
DC StartTime0 (0x990): 0x0000002cf101fb80

Note that 0x981 is left to 0 (mode Cyclic Acknowledge). Without that the system is in single shot which seems to be useless in your case (only one SYNC0 will be generated).

This configuration is done during the PREOP state (so after bus.init() in KickCAT).

Extract of the doc:
The cycle time of the SYNC0 signal is configured in the SYNC0 Cycle Time register (0x09A0:0x09A3),
the start time is set in the Start Time Cyclic Operation register (0x0990:0x0997). After the Sync Unit is
activated and the output of the SYNC0/1 signals is enabled (DC Activation register 0x0981), the Sync
Unit waits until the start time is reached and generates the first SYNC0 pulse.

9.2.3.3 Cyclic Acknowledge Mode
The Cyclic Acknowledge mode is typically used for generation of isochronous interrupts. The
acknowledged modes are selected by setting the Pulse Length of SYNC Signals to 0
(0x0982:0x0983). Each SyncSignal pulse remains active until it is acknowledged – typically by a
μController – by reading the appropriate SYNC0 or SYNC1 Status register (0x098E, 0x098F). The first
pulse is generated after the Start Time is reached, following pulses are generated when the next
regular SYNC0/1 event would occur.

[leducp]

Another info:

9.2.3.6 SyncSignal Initialization Example
The SyncSignal generation is initialized with the following procedure:

1. Enable DC SYNC Out Unit in ESC Configuration register (0x0141[2]=1; specific ESCs only)
2. Set SYNC/Latch PDI Configuration register (0x0151, initialized by SII EEPROM) to SYNC0/1
   output with appropriate driver settings.

It seems that as mention by @Rdk-T the eeprom contains a configuration that set the PDI in DC mode.

[leducp]

@thetooth any update about this problem? Since I don't have access to your hardware I cannot do much more to help you

[thetooth]

@leducp I tried quite a few things but never got anything satisfactory. I was only able to get the drive to move a handful of times by effectively "hot swapping" from codesys runtime over to kickcat, in this mode the drive will vibrate due to the lack of perfect synchronisation.

At least for these drives the timing requirements are extremely tight, the standard sleep function used in the ingenia example is non-functional and you must use timespec with drift compensation to align the control loop with the DC sync0 offset returned by the last device in the configuration(see here), this has an operating range of <50uS, that is if you write PDO data 25uS before or after sync0+toff then the drive discards the change (it's written to the drive register but not committed to the state machine) until the next time things happen to line up.

Additionally there are some gotchas with the way FRMW datagrams for the DC clock are handled, they must be part of the same frame as the PDO payload and also needs to be the first datagram, not doing this gives the same behaviour as above. I think we would have to modify kickcat a fair bit to get this happening.

Atm I am using the SOEM library which is working well enough. They're similar enough that if you give me a week or two I can likely setup my code to switch between kickcat and SOEM, that way if someone wants to have a stab at a DC implementation I'll be able to test, wouldn't mind implementing it myself but work would get in the way :(

[leducp]

@thetooth
Thank you for the feedback : it seems a more complicated yet coherent the way you describe the DC behavior.
Feel free to propose PRs if you have the time, I'll be happy to review them :)
Anyway your description will be really useful if I manage to put my hand on a DC compatible slave.