THREESCALE-2989 - simplify background deletion

[akostadinov]

What this PR does / why we need it:

Presently background deletion relies on Sidekiq batches which don't work well with ActiveJob. Specifically failing a job registers triggers callbacks for successful job if I remember correctly.

Moreover there was a bug that calling #delete was calling a state machine method instead of the activerecord method.

These two have the potential to cause indefinite rescheduling and multiplication of delete jobs eventually exhausting server performance and sidekiq redis memory.

With the new implementation, we only use a single ActiveRecord job, do not rely on Sidekiq batches and basically assuring order of operations, the logic can be much better understood. Runoffs are hopefully impossible but should be much easier to contain because now only the :deletion queue is used and stopping this queue should also freeze the problem until it can be properly debugged and fixed.

Which issue(s) this PR fixes

https://issues.redhat.com/browse/THREESCALE-2989

Verification steps

1. create a big tenant with many users, services, etc.
2. run DeleteObjectHierarchyWorker.delete_later(myprovider)
3. verify all objects are gone

For upgrade

1. schedule a big tenant deletion
2. stop server and sidekiq
3. upgrade
4. start server and sidekiq
5. verify all objects get deleted

Special notes for your reviewer:
This is just an early preview to gather feedback. It is very much untested in most regards except for a simple new test.

As for review, I think you better initially ignore old code and just read the new implementation of the Worker as new code. Then see if there are areas in the old code that have not been addressed by the new code. I know it is a pain.

It remains to test compatibility, self-rescheduling, other existing users of the worker (except for the service deletion which is the only one tested atm) and who knows what else.

[akostadinov]

There is one difference with previous logic. Previously #destroy without ! was used when destroyed by association. But I don't see why should this be done and so far haven't observed any negative effects. I think it is correct that the job should fail/retry if the #destroy! method failed.

[akostadinov]

Nice that these two are highlighted. Reading the original logic, all the reason for separate DeleteAccountHierarchyWorker and DeletePaymentSettingHierarchyWorker appears to be enforcing and order of executing the deletions. Since we do deleting serially here, the order is enforced by the order of entries put in background_deletion so I removed all that and just arranged the entries like this.

I must say that I cannot definitively claim that I understood all implications of the original code. More like I vaguely understood it. I found it more important to understand the intentions of the original authors because the code was misbehaving anyways and is heavily been refactored.

Although further testing may lead to a realization of details that I missed. So don't blindly trust me!