Comprehensive category filtering

[bedeho]

Scope:

• Add config for whitelisted/blacklisted categories (whitelist takes precedence)
• Add filtering based on that config (see Ability to filter out content as gateway operator #3037)
• Filter categories displayed in Discover view

Background

We will soon start having different instances of Atlas, with distinct branding. It would be good if there was some way we could allow these instances to brand them selves a little bit more strongly out of the gate in terms of the content they surfaced as well. This is the simplest way they can start to differentiate in a way which does rely on building any new product features, but instead at the content level. A second reason this came to mid was simply by observing how hopelessly bad our current category selection is. I have already inquired about addressing this through this issue #1659.

Proposal

Make it possible to configure each Atlas instance to only surface content associated with videos from a predefined set of video categories. By screening, I here mean hiding everywhere:

• Player: More from x/next video, also block playback of video not with suitable category.
• featured content
• search results
• channel view
• NFT marketplace
• discovery: Here also allow custom preview assets for each category, as suggested by Idea: Make it super easy for Atlas operator/community to manage set of categories in Discover view #1659. Arguably, the discovery section only needs to really show a subset of the full set of categories
• New/Channels ...
• Notifications
• etc.

Question

• Are there any parts of the product such filtering is difficult or not feasible because of missing Hydra filtering on video categories
• Are there any unintended edge cases or weird problems by trying to segment out only specific kinds of content in Atlas.

┆Issue is synchronized with this Asana task by Unito

[bedeho]

Addendum

I think for this to really be potent we would change the following

• Any member, not working group curators, can create categories. This is important for easy, nocode and permissionless deployment of new experimental instances that are differentiated in terms of new content verticals.
• Each Atlas instance filters on a set of these categories, X, globally as configured on build time.
• Publishing in this same instance requires user to select among X only.
• The discovery screen shows a set of display categories Y, where the operator for each category in this set defines
  • a graphic
  • a display name
  • a subset of X with at least one video category. (its fine for different display categories to include the same video category)
    so importantly Y != X.

The idea here is to make it super easy for a new operator to easily build a new video site focused on a specific type of content and community, without having to get permission from curators, without disrupting other instances and without having to be a developer.

[kdembler]

I would like to better understand what a display category is. Are those kept by the QN as well or do you imagine those being defined purely on the app (operator) level?

Secondly, I'm a bit skeptical of the "let's allow anyone to create a category" part. It will be cheap (it's metaprotocol) to spam QN with thousands of fake categories. This got me thinking a bit, do we even need categories stored in the QN? If we want to put more emphasis on operators pre-selecting categories and we also want to allow anyone to create them, why not make them a concept existing only on the app level? Each instance could have its own config for categories - map from ID to name, icon, image, etc. When publishing, you only specify a category ID, and QN would only work based on those IDs, it wouldn't understand what each category means, that would be resolved only inside the app. Runtime doesn't rely on categories in any way anyway. This way, operators are free to fully manage the categories they display, including display name etc. WDYT?

[bedeho]

Are those kept by the QN as well

No, they are purely application specific. Their purpose is to allow an independent and higher level of abstraction to the categories at the lower level. The main reason for this is because you can imagine the lowest level category space could encode very complex or granular notions about the content that are not ideal for organizing content in discovery. In this case, the user would probably not select the low level category either, it would be done automatically by the application, e.g. by having separate publishing flows, or automated detection on the media level.

Secondly, I'm a bit skeptical of the "let's allow anyone to create a category" part. It will be cheap (it's metaprotocol) to spam QN with thousands of fake categories.

I don't think this is a particular source of spam risk, as distinct from everything else. Anyone could spam comments or any number of the other metaprotocol messages we have. It would only be a problem if the app developer decided to expressionlessly present and use the full set of categories in their app, as if it was permission ed, which would be ill advised, and is not part of what I have in mind. The operator has to actively chose.

This got me thinking a bit, do we even need categories stored in the QN? If we want to put more emphasis on operators pre-selecting categories and we also want to allow anyone to create them, why not make them a concept existing only on the app level?

This is a fair point, and it is a fundamental question. I think the reason to have things public if it can plausibly be productive to have as building the network effect across apps, which is a core property we want to enable. I think this applies to this category information, as it helps different apps to present and build the pool of content in a collaborative way.

Now in the longer term, I think I think each app operator cannot blindly replicate all content from one category, because you don't know to what extent others are respecting these semantics, but this information about what content from a category should a given app use and present is something that adds no real value in being public. This state would live in Orion for example. Obviously, any content published via a given app would presumably automatically be presented in that same app immediately after whatever automated validation has been done.

Longer term, one way to make this collaboration more effective, while protecting the integrity of the experience in each app, is that whenever someone publishes to the content directory, the signature of whatever app they used can be included, so other apps at least know they can sort of trust the content if they have some trust in that applications validation. This would prevent some random person from starting a script and just publishing white noise content under a given category, which then other apps would either have to trustlessly replicate and display, or be really slow to manually confirm. Obviously other inputs can be used in the policy of an app, such as who the publisher is.

Each instance could have its own config for categories - map from ID to name, icon, image, etc. When publishing, you only specify a category ID, and QN would only work based on those IDs, it wouldn't understand what each category means, that would be resolved only inside the app.

I did not really understand this one.

[kdembler]

I think the reason to have things public if it can plausibly be productive to have as building the network effect across apps, which is a core property we want to enable. I think this applies to this category information, as it helps different apps to present and build the pool of content in a collaborative way.

Yeah I was considering this as well when writing my response, but my conclusion was that it actually doesn't help you much if the directory of categories is public - assuming writing to it is permissionless, even if you want to start a new instance and only include content from "Gaming" categories, there will likely be hundreds of those. So some amount of social coordination between operators to agree upon which categories are "genuine" will be required either way. It also wouldn't be problematic to look up what categories other instances are using and use the same in your config. There could be some directory managed by bigger operators to agree upon meaning behind category IDs. I think this would be needed regardless of whether we have on-chain categories or no.

One way to make this long term collaboration more effective, while protecting the integrity of the experience in each app, is that whenever someone publishes to the content directory, the signature of whatever app they used can be included, so other apps at least know they can sort of trust the content if they have some trust in that applications validation. This would prevent some random person from starting a script and just publishing white noise content under a given category, which then other apps would either have to trustlessly replicate and display, or be really slow to manually confirm. Obviously other inputs can be used in the policy of an app, such as who the publisher is.

Very good point.

[bedeho]

it actually doesn't help you much if the directory of categories is public - assuming writing to it is permissionless, even if you want to start a new instance and only include content from "Gaming" categories, there will likely be hundreds of those

I don't see how you get your conclusion here. The fact that there will be social coordination on semantics of categories does not negate the utility of having all content publicly labelled? I think we probably should discuss it later then ;)

[bedeho]

Very good point.

Obviously, same technique makes perfect sense for channels also.

[dmtrjsg]

This is a somewhat technical convo going on here, and I am 100 sure you guys are arriving to the best infrastructure possible to support the vision.

Also its getting really hot in my flat now so forgive me if below is captain obvious, or irrelevant.

As a gateway operator I'd be interested in having access to easily navigable content dir, that is already curated by DAO content WG so has some degree of MECE to it. Bedeho referred to it as L1. So I'd be interested in visualising them differently, having an opportunity to present single L1 category in multiple L2; multiple L1 in a single L2; exclude some L1s and add custom L2s. As per the aspect of indexable categories from/by other GW operators, would be cool to be able to access it, so a signature of source may play out nicely not only from perspective of traceability. Being able to remap other GW L2s to my own GW L2s would help I would think. And doing the changes to GW apps without the need to rely on any sort of review/ validation process is super desirable.

1. Having some default catalog easy to get started with
2. Flexibility of adding your presentation of existing catalogue
3. Ability to add your own categories/ without need to collaborate with other GW operators or any sort of reviews by DAO Content WG

[kdembler]

Just to jot down our agreement on the call:

1. Anyone can create a category
2. Categories have some very basic metadata - title, optional description, optional parent category
3. QN needs to handle unknown categories. If I create a video with category ID 111123123123151, QN needs to properly index that video and just not offer any additional metadata for it
4. We create display categories inside the app and overwrite any on-chain category metadata
5. Are categories editable by their creators? That's the only thing I think we haven't settled on

[bedeho]

Thanks for summary!

Are categories editable by their creators? That's the only thing I think we haven't settled on

Lets not for now. No editing, no deleting.

[toiletgranny]

OK, so I had @kdembler introduce me to this subject earlier today, and — with my understanding — no additional designs are necessary for this issue (excluding #3213) on top of what's already designed and implemented (e.g. the Discover page, the Video category page, the dropdown for selecting a Video category in the Video workspace, etc.).

---

There's, however, one more product/design matter that I'd like to suggest reconsidering.

@bedeho, as per your addendum:

Publishing in this same instance requires user to select among X only.

If my understanding is correct, the UX drawback of this would be that as a video creator, while picking a video category from X, I would not know how my choice is later translated into Y. Of course, one simple remedy for this would be adjusting this part of the UI (see the screenshot above) to include some sort of video categories conversion table or displaying Atlas instance display categories (Y) as captions tied to each of the Joystream categories (X). But, to me, this seems overly excessive for something as simple as selecting a category for my video, resulting in equally poor UX. My thesis is that, as a video creator, at this point, I just want to pick a video category and move on with uploading my video, and not be bothered with how my choice now will be later represented in other potential instances of this video platform.

For this reason, I'd recommend letting the user choose from the list of display categories native to the Atlas instance (Y). Simply. What I see is what I get.

But for this to work, due to the interchangeable nature of display categories (Y) and their relation to Joystream categories (X), to ensure proper categories mapping across different instances of Atlas, the Atlas instance operator must specify one key Joystream category to which each display category should be mapped so that we always have a one-to-one relation, no matter which display category the video creator selects during upload.

[bedeho]

For this reason, I'd recommend letting the user choose from the list of display categories native to the Atlas instance (Y). Simply. What I see is what I get. ...

I get the problem here, but such a mapping seems quite arbitrary to me, and it's hard to really anticipate how bad the UX problem is really going to be, because it's not clear how big the discrepancy between X & Y will in the end be, nor how genuinely confusing the X category space will really be, it will after all be more granular, so you could make the converse point that it will be easier to find a suitable space for your video. Honestly, I think its hard to say, so the path of least resistance seems like the right option right now, and then when we see how these category spaces are used, and how creators get on with publishing (outside of youtube synch), then the question can be reevaluated with the benefit of more objective information.

[toiletgranny]

Alright, let's do that!

@dmtrjsg, assigning the issue back to you since it requires no additional designs.

[kdembler]

@toiletgranny What about category information in Video view details? It can now be multiple display categories

[bedeho]

What about category information in Video view details? It can now be multiple display categories

Why would that be the case?

[kdembler]

If a video belongs to a Joystream category X1, that category can be included in multiple display categories, right? We probably should only display display categories in the app, so for a given video we may need to show a list of display categories X1 belongs to

[toiletgranny]

Right, of course, @kdembler! And since we agreed to make the creator select from Joystream categories during the video upload process, that video might end up being assigned to multiple display categories.

Another thing I realized seeing Klaudiusz's comment and having talked with Kuba about it yesterday is that:

1. Say there's an instance of Atlas that's all about music. Let's call it "Joysound". It only fetches Joystream videos from the "Music" category.
2. On "Joysound", there's a set of display categories with different music genres, like "Jazz", "Funk", "Pop", etc. These are all connected to the "Music" Joystream category.
3. As a video creator trying to upload a video via "Joysound", I can only specify that this video should belong to the "Music" category. I don't have an option to specify which genre specifically.
4. And, to make things worse, having uploaded such a video, on the video page it will be listed under literally all display categories "Joysound" has to offer.

I can see two ways out of here:

1. Let's revisit the idea I described previously, which is to let video creators specify the display category they want their video to be put in, rather than having them select from the list of Joystream categories.
2. On the Video page, let's keep displaying Joystream categories, rather than display categories. This, however, creates a discrepancy for viewers, too. First, they discover a video via a display category page, but when go into that video's page, it's listed under a different category. Not ideal.

[kdembler]

@toiletgranny I think in your example, this is problematic indeed, but I don't think it's a systemic issue, rather with the specific setup you mentioned. In real life, the operator wouldn't create multiple display categories all including a single Joystream category, but rather have a separate Joystream category for each music genre. That's where the part with everybody being able to create new categories comes in. The operator may even go further and break down the categories even further, so that their Jazz display category contains more specific Joystream categories, like Acid jazz and Jazz rap

[toiletgranny]

Oh, I see. Right, that makes sense. Thank you for clarifying that, @kdembler.

Based on our discussion, I updated the video page designs to support displaying more than one video category. Designs, docs, and changelog can be found here.

I can't think of any other implication of this display categories idea that would require design changes. However, if you find something, please feel free to move the issue back to the design column!

[kdembler]

All right, thanks for taking a look, will keep you updated