Investments: RuntimeApi for redeem and collected tokens

[lemunozm]

Description

Output from a conversation with @onnovisser:

Apps needs an improvement in Investment API to obtain the following:

• remaining redeem tokens for the whole portfolio on an account
• collectable invest tokens for the whole portfolio of an account

[wischli]

@onnovisser Could you elaborate on the desired inputs? Would it just be an AccountId or can it include additional parameters such as PoolId?

collectable invest tokens for the whole portfolio of an account

Just to be clear, this refers to the amount of tranche tokens which can be collected by an account after their investment was processed during epoch execution? AFAIK, this information can only be retrieved by simulating a collection. cc @mustermeiszer

[onnovisser]

@wischli
We want to display the total investment value of an account. For this we need, for each tranche, the amount of tranche tokens that the account "owns". What we see as owned is:

• The regular tranche token balance of an account
• The tokens that can be collected after epoch execution
• The part of the tokens of a redeem order that haven't yet been redeemed before/after epoch execution
• Staked tokens

[wischli]

@wischli We want to display the total investment value of an account. For this we need, for each tranche, the amount of tranche tokens that the account "owns". What we see as owned is:

• The regular tranche token balance of an account
• The tokens that can be collected after epoch execution
• The part of the tokens of a redeem order that haven't yet been redeemed before/after epoch execution
• Staked tokens

Thanks for the details. I would still like to know the expected input parameters for the runtime API call. In the initial issue description it appears to be just an AccountId. Based on your response, it appears to be (AccountId, TrancheId). The latter is much much simpler and more lightweight on-chain. So hoping for this one.

[onnovisser]

just AccountId would be ideal. The issue with AccountId, TrancheId is that we would first need to know for which tranches the user has any investment. Otherwise we'd be forced to call it for every tranche of every pool.

[wischli]

I actually think that only the following requirement should be added as runtime api:

• The tokens that can be collected after epoch execution

The rest can be obtained in a more lightweight fashion by querying entries of the corresponding double storage maps. AFAIK, querying api.query.someDoubleMap.entries() via RPC counts as single read. In contrast, if we iterated all keys on the chain and filtered for the provided investor account id, we have as many reads as there are keys in the map. This would be a super heavy runtime API call!

I was wondering whether Apps are using any indexer service? If so, seems like a great use case.

Here's how you'd derive the other three above requirements. Code works but can be optimized by utilizing reduce of course.

• The regular tranche token balance of an account

const trancheIdsWithAmounts = (await api.query.ormlTokens.accounts.entries())
  .map(([storageKey, amount]) => [...storageKey.toHuman(), amount])
  .filter(([accountId, asset, amount]) => accountId === investor && asset.Tranche)
  .map(([_, investmentId, amount]) => [investmentId.Tranche[1], amount])

• The part of the tokens of a redeem order that haven't yet been redeemed before/after epoch execution

const redeemInvestmentIdsWithAmounts = (await api.query.investments.redeemOrders.entries())
  .map(([storageKey, amount]) => [...storageKey.toHuman(), amount])
  .filter(([accountId, investmentId, amount]) => accountId === investor)
  .map(([_, investmentId, amount]) => [investmentId, amount])

The part of tokens of an invest order that have not been processed yet:

const investInvestmentIdsWithAmounts = (await api.query.investments.redeemOrders.entries())
  .map(([storageKey, amount]) => [...storageKey.toHuman(), amount])
  .filter(([accountId, investmentId, amount]) => accountId === investor)
  .map(([_, investmentId, amount]) => [investmentId, amount])

• Staked tokens

Via rewardsApi.listCurrencies

WDYT @onnovisser ?

[onnovisser]

@wischli
yes the staked tokens we can easily get otherwise, that's okay. The balances too.
However, the remaining redeem I would include with the collectable invest, as they are closely related. Maybe my explanation of the redeem wasn't clear. I meant the remaining tokens that still have to be redeemed after epoch execution.
Here's how we currently get the collectable invest / remaining redeem for a single tranche: https://github.com/centrifuge/apps/blob/main/centrifuge-js/src/modules/pools.ts#L2738

[onnovisser]

Maybe like you said, using subquery for this would be more practical, if it's not feasible to do via a Runtime API

[wischli]

@wischli yes the staked tokens we can easily get otherwise, that's okay. The balances too. However, the remaining redeem I would include with the collectable invest, as they are closely related. Maybe my explanation of the redeem wasn't clear. I meant the remaining tokens that still have to be redeemed after epoch execution. Here's how we currently get the collectable invest / remaining redeem for a single tranche: https://github.com/centrifuge/apps/blob/main/centrifuge-js/src/modules/pools.ts#L2738

It appears to me that your current flow already has knowledge of all remaining tokens that still have to be redeemed after epoch execution?! You only need to feed it all corresponding pool and tranche ids for the investor. As mentioned: This should be solved off-chain IMO.

[onnovisser]

Yes, Ideally we wouldn't even have that function in our frontend though. I just copied how the chain computes the collectable tokens. I think Frederik is also of the opinion that most data we get should be done via runtime calls if possible, to also be more resilient to chain changes. But if there's no feasible way then we can do it in the frontend.

Btw, what is the investments portfolio currently returning? I see here it gets Accountant::balance is this just the ormlTokens balance? https://github.com/centrifuge/centrifuge-chain/blob/main/pallets/investments/src/lib.rs#L1117

[wischli]

Decision from sync call: The Runtime API exposes a call which returns the investment portfolio of an account:

fn account_portfolio(investor: account_id) -> Vec<(InvestmentId, InvestmentPortfolio)>;

type InvestmentId = TrancheCurrency;
pub struct TrancheCurrency {
	pub pool_id: PoolId,
	pub tranche_id: TrancheId,
}

struct InvestmentPortfolio {
	/// The unprocessed invest order amount in pool currency
	pending_invest_currency: Balance,
	/// The amount of tranche tokens which can be collected for an invest order
	claimable_tranche_tokens: Balance,
	/// The amount of tranche tokens which can be transferred
	free_tranche_tokens: Balance,
	/// The amount of tranche tokens which cannot be transferred
	locked_tranche_tokens_: Balance,
	/// The unprocessed redeem order amount in tranche tokens
	pending_redeem_tranche_tokens: Balance,
	/// The amount of pool currency which can be collected for a redeem order
	claimable_currency: Balance
}

cc @onnovisser

EDIT: Changed AccountPortfolio to InvestmentPortfolio.

[mustermeiszer]

Nit: would name it InvestmentPortfolio

[lemunozm]

Makes sense since we have another portfolio related to an account for loans

[wischli]

IMO the above sketch of the InvestmentPortfolio is flawed in that it does not offer currency granularity. Instead of resolving each field to Balance, it should be Map<CurrencyId, Balance>.

For instance, I don't think all tranche tokens can be treated equally? Moreover, assuming that we have different pool currencies in the future, these might neither have the same demonination nor FIAT value.

pub struct InvestmentPortfolio<Balance, CurrencyId> {
	/// The unprocessed invest order amount in pool currency
	pub pending_invest_currency: BTreeMap<CurrencyId, Balance>,
	/// The amount of tranche tokens which can be collected for an invest order
	pub claimable_tranche_tokens: BTreeMap<CurrencyId, Balance>,
	/// The amount of tranche tokens which can be transferred
	pub free_tranche_tokens: BTreeMap<CurrencyId, Balance>,
	/// The amount of tranche tokens which cannot be transferred
	pub frozen_tranche_tokens: BTreeMap<CurrencyId, Balance>,
	/// The unprocessed redeem order amount in tranche tokens
	pub pending_redeem_tranche_tokens: BTreeMap<CurrencyId, Balance>,
	/// The amount of pool currency which can be collected for a redeem order
	pub claimable_currency: BTreeMap<CurrencyId, Balance>,
}

WDYT about switching from Balance to Map<CurrencyId, Balance> @onnovisser @mustermeiszer @hieronx ?

[wischli]

@onnovisser I also saw that there already exists and InvestmentsApi.investmentPortfolio which returns a list of tuples that partially contains data of the new InvestmentPortfolio represented by this issue.

fn investment_portfolio(account_id: AccountId) -> Option<Vec<(PoolId, CurrencyId, InvestmentId, Balance)>>

Is this API used by Apps? Will it still be used when we have the new portfolio API?

[hieronx]

@wischli the return type of the API is Vec<(InvestmentId, InvestmentPortfolio), so one tuple item for each investment ID, therefore each investment portfolio struct is specific to 1 tranche, right? Each tranche has 1 tranche token and 1 currency (the pool currency). So I think the information is already included that way.

[mustermeiszer]

WDYT about switching from Balance to Map<CurrencyId, Balance> @onnovisser @mustermeiszer @hieronx ?
for tranches this is given in the InvestemtId the pool currency could be added IMO. Best to the InvestmentPortfolio struct.

[wischli]

@wischli the return type of the API is Vec<(InvestmentId, InvestmentPortfolio), so one tuple item for each investment ID, therefore each investment portfolio struct is specific to 1 tranche, right? Each tranche has 1 tranche token and 1 currency (the pool currency). So I think the information is already included that way.

Good point, I overlooked the return type of the fn account_portfolio 🙈.