Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

live migration: skip identical state transitions #679

Open
tmakatos opened this issue May 12, 2022 · 1 comment
Open

live migration: skip identical state transitions #679

tmakatos opened this issue May 12, 2022 · 1 comment
Labels
enhancement New feature or request

Comments

@tmakatos
Copy link
Member

@jlevon noticed that during LM QEMU might tell us to switch to the same migration state, here's an example from NVMf/vfio-user:

[2022-05-11 16:27:35.610634] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: migration: transitioning from state pre-copy to state stop-and-copy
[2022-05-11 16:27:35.610644] vfio_user.c:3510:vfio_user_migration_device_state_transition: *DEBUG*: /var/run controller state 3, migration state 2
[2022-05-11 16:27:35.610694] vfio_user.c:3031:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Dump SAVE
[2022-05-11 16:27:35.610732] vfio_user.c:3036:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Registers
[2022-05-11 16:27:35.610749] vfio_user.c:3037:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CSTS 0x1
[2022-05-11 16:27:35.610766] vfio_user.c:3038:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CAP  0x201e0100ff
[2022-05-11 16:27:35.610782] vfio_user.c:3039:vfio_user_ctrlr_dump_migr_data: *NOTICE*: VS   0x10300
[2022-05-11 16:27:35.610808] vfio_user.c:3040:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CC   0x460001
[2022-05-11 16:27:35.610835] vfio_user.c:3041:vfio_user_ctrlr_dump_migr_data: *NOTICE*: AQA  0xff003f
[2022-05-11 16:27:35.610859] vfio_user.c:3042:vfio_user_ctrlr_dump_migr_data: *NOTICE*: ASQ  0xbffdd000
[2022-05-11 16:27:35.610878] vfio_user.c:3043:vfio_user_ctrlr_dump_migr_data: *NOTICE*: ACQ  0xbffde000
[2022-05-11 16:27:35.610892] vfio_user.c:3045:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Number of IO Queues 1
[2022-05-11 16:27:35.610910] vfio_user.c:3059:vfio_user_ctrlr_dump_migr_data: *NOTICE*: sqid:0, bar0_doorbell:35
[2022-05-11 16:27:35.610934] vfio_user.c:3066:vfio_user_ctrlr_dump_migr_data: *NOTICE*: SQ sqid:0, cqid:0, sqhead:35, size:64, dma_addr:0xbffdd000
[2022-05-11 16:27:35.610952] vfio_user.c:3071:vfio_user_ctrlr_dump_migr_data: *NOTICE*: cqid:0, bar0_doorbell:35
[2022-05-11 16:27:35.610970] vfio_user.c:3078:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CQ cqid:0, phase:1, cqtail:35, size:256, iv:0, ien:1, dma_addr:0xbffde000
[2022-05-11 16:27:35.610986] vfio_user.c:3059:vfio_user_ctrlr_dump_migr_data: *NOTICE*: sqid:1, bar0_doorbell:1
[2022-05-11 16:27:35.611009] vfio_user.c:3066:vfio_user_ctrlr_dump_migr_data: *NOTICE*: SQ sqid:1, cqid:1, sqhead:1, size:256, dma_addr:0xbffd8000
[2022-05-11 16:27:35.611025] vfio_user.c:3071:vfio_user_ctrlr_dump_migr_data: *NOTICE*: cqid:1, bar0_doorbell:1
[2022-05-11 16:27:35.611040] vfio_user.c:3078:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CQ cqid:1, phase:1, cqtail:1, size:256, iv:0, ien:0, dma_addr:0xbffdc000
[2022-05-11 16:27:35.611059] vfio_user.c:3083:vfio_user_ctrlr_dump_migr_data: *NOTICE*: SAVE Dump Done
[2022-05-11 16:27:35.611078] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: migration: transitioned from state pre-copy to state stop-and-copy
[2022-05-11 16:27:35.611090] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: region9: wrote 0x2 to (0:4)
[2022-05-11 16:27:35.611106] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: device unquiesced
[2022-05-11 16:27:35.611116] vfio_user.c:2946:vfio_user_dev_quiesce_done: *DEBUG*: /var/run is in MIGRATION state
[2022-05-11 16:27:35.611568] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [(nil), 0xa0000), 0 dirty pages
[2022-05-11 16:27:35.612568] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0xc0000, 0xcb000), 0 dirty pages
[2022-05-11 16:27:35.613571] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0xcb000, 0xce000), 0 dirty pages
[2022-05-11 16:27:35.614565] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0xce000, 0xe8000), 0 dirty pages
[2022-05-11 16:27:35.615567] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0xe8000, 0xf0000), 0 dirty pages
[2022-05-11 16:27:35.616566] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0xf0000, 0x100000), 0 dirty pages
[2022-05-11 16:27:35.618360] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0x100000, 0xc0000000), 0 dirty pages
[2022-05-11 16:27:35.618821] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: dirty pages: get [0x100000000, 0x140000000), 0 dirty pages
[2022-05-11 16:27:35.619567] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: region9: read 0x2 from (0:4)
[2022-05-11 16:27:35.620565] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: quiescing device
[2022-05-11 16:27:35.620582] vfio_user.c:2979:vfio_user_dev_quiesce_cb: *DEBUG*: /var/run starts to quiesce
[2022-05-11 16:27:35.620591] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: device quiesced immediately
[2022-05-11 16:27:35.620602] vfio_user.c:2843:vfio_user_log: *DEBUG*: /var/run: migration: transitioning from state stop-and-copy to state stop-and-copy
[2022-05-11 16:27:35.620611] vfio_user.c:3510:vfio_user_migration_device_state_transition: *DEBUG*: /var/run controller state 5, migration state 2
[2022-05-11 16:27:35.620627] vfio_user.c:3031:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Dump SAVE
[2022-05-11 16:27:35.620651] vfio_user.c:3036:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Registers
[2022-05-11 16:27:35.620670] vfio_user.c:3037:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CSTS 0x1
[2022-05-11 16:27:35.620686] vfio_user.c:3038:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CAP  0x201e0100ff
[2022-05-11 16:27:35.620703] vfio_user.c:3039:vfio_user_ctrlr_dump_migr_data: *NOTICE*: VS   0x10300
[2022-05-11 16:27:35.620721] vfio_user.c:3040:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CC   0x460001
[2022-05-11 16:27:35.620738] vfio_user.c:3041:vfio_user_ctrlr_dump_migr_data: *NOTICE*: AQA  0xff003f
[2022-05-11 16:27:35.620753] vfio_user.c:3042:vfio_user_ctrlr_dump_migr_data: *NOTICE*: ASQ  0xbffdd000
[2022-05-11 16:27:35.620778] vfio_user.c:3043:vfio_user_ctrlr_dump_migr_data: *NOTICE*: ACQ  0xbffde000
[2022-05-11 16:27:35.620808] vfio_user.c:3045:vfio_user_ctrlr_dump_migr_data: *NOTICE*: Number of IO Queues 1
[2022-05-11 16:27:35.620835] vfio_user.c:3059:vfio_user_ctrlr_dump_migr_data: *NOTICE*: sqid:0, bar0_doorbell:35
[2022-05-11 16:27:35.620858] vfio_user.c:3066:vfio_user_ctrlr_dump_migr_data: *NOTICE*: SQ sqid:0, cqid:0, sqhead:35, size:64, dma_addr:0xbffdd000
[2022-05-11 16:27:35.620874] vfio_user.c:3071:vfio_user_ctrlr_dump_migr_data: *NOTICE*: cqid:0, bar0_doorbell:35
[2022-05-11 16:27:35.620892] vfio_user.c:3078:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CQ cqid:0, phase:1, cqtail:35, size:256, iv:0, ien:1, dma_addr:0xbffde000
[2022-05-11 16:27:35.620907] vfio_user.c:3059:vfio_user_ctrlr_dump_migr_data: *NOTICE*: sqid:1, bar0_doorbell:1
[2022-05-11 16:27:35.620931] vfio_user.c:3066:vfio_user_ctrlr_dump_migr_data: *NOTICE*: SQ sqid:1, cqid:1, sqhead:1, size:256, dma_addr:0xbffd8000
[2022-05-11 16:27:35.620954] vfio_user.c:3071:vfio_user_ctrlr_dump_migr_data: *NOTICE*: cqid:1, bar0_doorbell:1
[2022-05-11 16:27:35.620977] vfio_user.c:3078:vfio_user_ctrlr_dump_migr_data: *NOTICE*: CQ cqid:1, phase:1, cqtail:1, size:256, iv:0, ien:0, dma_addr:0xbffdc000

That's a bit weird, not sure why QEMU does this. @jraman567 @john-johnson-git any ideas?
Irrespecively, we should skip this in libvfio-user.
@tmakatos tmakatos added the enhancement New feature or request label May 12, 2022
@jraman567
Copy link
Contributor

@tmakatos I do recall receiving transitions from & to the same state - I skipped such requests in the server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tmakatos @jraman567