Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: export tagged content library as csv [FC-0049] #34246

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
52bc665
feat: export tagged course as csv
rpenido Jan 25, 2024
f275908
docs: add comment
rpenido Jan 25, 2024
a870dfd
fix: add select_related to ObjectTag query
rpenido Jan 26, 2024
4dd027a
fix: always use objecttag.value
rpenido Jan 26, 2024
5fb03aa
docs: change comment position
rpenido Jan 26, 2024
8f1238e
refactor: rename serializer to a more sane name
rpenido Jan 26, 2024
bdedf93
refactor: remove download param
rpenido Jan 26, 2024
04ca072
refactor: create a new view to export objecttags
rpenido Jan 26, 2024
a8a6e7e
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Jan 26, 2024
b01d6d4
refactor: change api and view structure
rpenido Jan 29, 2024
0871537
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Jan 29, 2024
18a8425
docs: remove old comment
rpenido Jan 29, 2024
38ae353
docs: revert view docstring
rpenido Jan 30, 2024
b65a6c8
fix: remove include_children query param
rpenido Jan 30, 2024
f5ac3a6
fix: removing .all() call from queryset
rpenido Jan 30, 2024
debf254
refactor: method rename
rpenido Jan 30, 2024
9726d6d
fix: filter deleted tags
rpenido Jan 30, 2024
c8c12bb
fix: pylint
rpenido Jan 30, 2024
1f11b17
fix: quote string in csv export
rpenido Jan 30, 2024
484c042
test: add querycount
rpenido Jan 30, 2024
6b6ba34
fix: pylint
rpenido Jan 30, 2024
9096454
fix: pylint..
rpenido Jan 30, 2024
a688689
fix: pylint
rpenido Jan 30, 2024
30e06d0
test: compare results to hardcoded strings
pomegranited Jan 30, 2024
35a3d2b
test: Adds "deleted" object tags to ensure they are omitted from results
pomegranited Jan 30, 2024
1e13f54
test: adds untagged blocks with children
pomegranited Jan 30, 2024
e9335c8
revert: undo removed property
rpenido Jan 30, 2024
ab1a69e
style: fix camelCase
rpenido Jan 30, 2024
db9116d
refactor: remove xblock from TaggedContent and include_children param
rpenido Jan 30, 2024
9b3dee8
fix: remove UsageKey
rpenido Jan 30, 2024
821e216
fix: removing unused import
rpenido Jan 31, 2024
6207915
refactor: cleaning code
rpenido Jan 31, 2024
35bc860
test: refactors tests so shared data can be re-used
pomegranited Jan 31, 2024
7a28742
refactor: refactoring api, helper and view code
rpenido Jan 31, 2024
fabb729
docs: add comment about ObjectTag query
rpenido Jan 31, 2024
548d57c
test: use CourseFactory and BlockFactory
pomegranited Jan 30, 2024
a1d41fd
test: fix variable name
rpenido Feb 1, 2024
f07b841
fix: delete unwanted file
rpenido Feb 1, 2024
233135a
test: fix query count
rpenido Feb 1, 2024
ac98812
test: fix expected value with new tags
rpenido Feb 1, 2024
309ce94
fix: use variables from outer function
rpenido Feb 1, 2024
32cdf93
test: use UserFactory
rpenido Feb 1, 2024
4ed7570
style: removed unused imports
rpenido Feb 1, 2024
ee5bc3d
chore: trigger CI
rpenido Feb 1, 2024
4bc8ca7
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Feb 2, 2024
6d4c23a
fix: disable default staff user from module store mixin
rpenido Feb 2, 2024
da3fdf9
style: fix case
rpenido Feb 5, 2024
726b7ef
Merge branch 'jill/rpenido/fal-3610-download-course-tag-spreadsheet' …
rpenido Feb 6, 2024
fd5a542
docs: adds removed docstring
rpenido Feb 6, 2024
f84abd0
feat: export tagged library as csv
rpenido Feb 6, 2024
79a1786
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Feb 8, 2024
dfe43be
fix: cleaning merged code
rpenido Feb 8, 2024
5245264
style: run isort
rpenido Feb 8, 2024
06810a2
Merge branch 'rpenido/fal-3610-download-course-tag-spreadsheet' into …
rpenido Feb 9, 2024
c82e9cb
refactor: use taxonomy.export_id in header
rpenido Feb 9, 2024
f750076
Merge branch 'rpenido/fal-3610-download-course-tag-spreadsheet' into …
rpenido Feb 10, 2024
01b9b5f
refactor: change `object_id` to `context_id`
rpenido Feb 15, 2024
f87fc4c
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Feb 15, 2024
779cc98
chore: trigger CI
rpenido Feb 15, 2024
f637076
Merge branch 'rpenido/fal-3610-download-course-tag-spreadsheet' into …
rpenido Feb 15, 2024
3cf462f
fix: rebasing
rpenido Feb 15, 2024
aa93510
fix: rename variable to fix typing
rpenido Feb 15, 2024
e94131f
fix: pylint
rpenido Feb 15, 2024
4a3d092
Merge branch 'master' into rpenido/fal-3610-download-course-tag-sprea…
rpenido Feb 15, 2024
9a1729d
Merge branch 'rpenido/fal-3610-download-course-tag-spreadsheet' into …
rpenido Feb 15, 2024
6518a91
refactor: changing condition checking
rpenido Feb 15, 2024
8ddf875
fix: check invalidkey
rpenido Feb 15, 2024
ed547be
fix: better error handling
rpenido Feb 16, 2024
df06d9a
docs: reverting unintended change
rpenido Feb 16, 2024
a153ec9
refactor: change functions to private
rpenido Feb 16, 2024
610c01a
refactor: cleaning build_object_tree_with_objecttags function
rpenido Feb 16, 2024
4012481
fix: explicit typing
rpenido Feb 16, 2024
4ad87c2
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 16, 2024
5caf952
fix: cleaning some types
rpenido Feb 16, 2024
df13345
docs: fix comment
rpenido Feb 16, 2024
e6a0dce
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 16, 2024
aa38f08
fix: typo
rpenido Feb 20, 2024
805ed98
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 21, 2024
0fd512e
docs: improve docstring
rpenido Feb 23, 2024
11d6de2
fix: remove uwsgi
rpenido Feb 23, 2024
632556c
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 23, 2024
83b16cc
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 27, 2024
6f02755
fix: update code with upstream changes
rpenido Feb 27, 2024
19c86a6
fix: removing unused import
rpenido Feb 27, 2024
6189a9f
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 28, 2024
07e8fa9
Merge branch 'master' into rpenido/fal-3611-download-library-tag-spre…
rpenido Feb 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 13 additions & 8 deletions openedx/core/djangoapps/content_tagging/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@

import openedx_tagging.core.tagging.api as oel_tagging
from django.db.models import Exists, OuterRef, Q, QuerySet
from opaque_keys.edx.keys import CourseKey, LearningContextKey
from opaque_keys.edx.keys import CourseKey
from opaque_keys.edx.locator import LibraryLocatorV2
from openedx_tagging.core.tagging.models import ObjectTag, Taxonomy
from organizations.models import Organization

Expand Down Expand Up @@ -130,24 +131,28 @@ def get_unassigned_taxonomies(enabled=True) -> QuerySet:


def get_all_object_tags(
content_key: LearningContextKey,
content_key: LibraryLocatorV2 | CourseKey,
) -> tuple[ObjectTagByObjectIdDict, TaxonomyDict]:
"""
Get all the object tags applied to components in the given course/library.

Includes any tags applied to the course/library as a whole.
Returns a tuple with a dictionary of grouped object tags for all blocks and a dictionary of taxonomies.
rpenido marked this conversation as resolved.
Show resolved Hide resolved
"""
# ToDo: Add support for other content types (like LibraryContent and LibraryBlock)
context_key_str = str(content_key)
# We use a block_id_prefix (i.e. the modified course id) to get the tags for the children of the Content
# (course/library) in a single db query.
if isinstance(content_key, CourseKey):
course_key_str = str(content_key)
# We use a block_id_prefix (i.e. the modified course id) to get the tags for the children of the Content
# (course) in a single db query.
block_id_prefix = course_key_str.replace("course-v1:", "block-v1:", 1)
block_id_prefix = context_key_str.replace("course-v1:", "block-v1:", 1)
elif isinstance(content_key, LibraryLocatorV2):
block_id_prefix = context_key_str.replace("lib:", "lb:", 1)
else:
raise NotImplementedError(f"Invalid content_key: {type(content_key)} -> {content_key}")

# There is no API method in oel_tagging.api that does this yet,
# so for now we have to build the ORM query directly.
all_object_tags = list(ObjectTag.objects.filter(
Q(object_id__startswith=block_id_prefix) | Q(object_id=course_key_str),
Q(object_id__startswith=block_id_prefix) | Q(object_id=content_key),
Q(tag__isnull=False, tag__taxonomy__isnull=False),
).select_related("tag__taxonomy"))

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,15 @@

from __future__ import annotations

from typing import Iterator
from typing import Any, Callable, Iterator, Union

from attrs import define
from opaque_keys.edx.keys import CourseKey, LearningContextKey
from opaque_keys.edx.keys import CourseKey, UsageKey
from opaque_keys.edx.locator import LibraryLocatorV2
from xblock.core import XBlock

import openedx.core.djangoapps.content_libraries.api as library_api
from openedx.core.djangoapps.content_libraries.api import LibraryXBlockMetadata
from xmodule.modulestore.django import modulestore

from ...types import ObjectTagByObjectIdDict, ObjectTagByTaxonomyIdDict
Expand Down Expand Up @@ -38,51 +42,137 @@ def iterate_with_level(
yield from iterate_with_level(child, level + 1)


def build_object_tree_with_objecttags(
content_key: LearningContextKey,
object_tag_cache: ObjectTagByObjectIdDict,
) -> TaggedContent:
def _get_course_tagged_object_and_children(
course_key: CourseKey, object_tag_cache: ObjectTagByObjectIdDict
) -> tuple[TaggedContent, list[XBlock]]:
"""
Returns the object with the tags associated with it.
Returns a TaggedContent with course metadata with its tags, and its children.
"""
store = modulestore()

if isinstance(content_key, CourseKey):
course = store.get_course(content_key)
if course is None:
raise ValueError(f"Course not found: {content_key}")
else:
raise NotImplementedError(f"Invalid content_key: {type(content_key)} -> {content_key}")
course = store.get_course(course_key)
if course is None:
raise ValueError(f"Course not found: {course_key}")

display_name = course.display_name_with_default
course_id = str(course.id)
course_id = str(course_key)

tagged_course = TaggedContent(
display_name=display_name,
display_name=course.display_name_with_default,
block_id=course_id,
category=course.category,
object_tags=object_tag_cache.get(str(content_key), {}),
object_tags=object_tag_cache.get(course_id, {}),
children=None,
)

return tagged_course, course.children if course.has_children else []


def _get_library_tagged_object_and_children(
library_key: LibraryLocatorV2, object_tag_cache: ObjectTagByObjectIdDict
) -> tuple[TaggedContent, list[LibraryXBlockMetadata]]:
"""
Returns a TaggedContent with library metadata with its tags, and its children.
"""
library = library_api.get_library(library_key)
if library is None:
raise ValueError(f"Library not found: {library_key}")

library_id = str(library_key)

tagged_library = TaggedContent(
display_name=library.title,
block_id=library_id,
category='library',
object_tags=object_tag_cache.get(library_id, {}),
children=None,
)

library_components = library_api.get_library_components(library_key)
children = [
LibraryXBlockMetadata.from_component(library_key, component)
for component in library_components
]

return tagged_library, children


def _get_xblock_tagged_object_and_children(
usage_key: UsageKey, object_tag_cache: ObjectTagByObjectIdDict
) -> tuple[TaggedContent, list[XBlock]]:
"""
Returns a TaggedContent with xblock metadata with its tags, and its children.
"""
store = modulestore()
block = store.get_item(usage_key)
block_id = str(usage_key)
tagged_block = TaggedContent(
display_name=block.display_name_with_default,
block_id=block_id,
category=block.category,
object_tags=object_tag_cache.get(block_id, {}),
children=None,
)

blocks = [(tagged_course, course)]
return tagged_block, block.children if block.has_children else []


def _get_library_block_tagged_object(
library_block: LibraryXBlockMetadata, object_tag_cache: ObjectTagByObjectIdDict
) -> tuple[TaggedContent, None]:
"""
Returns a TaggedContent with library content block metadata and its tags,
and 'None' as children.
"""
block_id = str(library_block.usage_key)
tagged_library_block = TaggedContent(
display_name=library_block.display_name,
block_id=block_id,
category=library_block.usage_key.block_type,
object_tags=object_tag_cache.get(block_id, {}),
children=None,
)

return tagged_library_block, None


def build_object_tree_with_objecttags(
content_key: LibraryLocatorV2 | CourseKey,
object_tag_cache: ObjectTagByObjectIdDict,
) -> TaggedContent:
"""
Returns the object with the tags associated with it.
"""
get_tagged_children: Union[
# _get_course_tagged_object_and_children type
Callable[[LibraryXBlockMetadata, dict[str, dict[int, list[Any]]]], tuple[TaggedContent, None]],
# _get_library_block_tagged_object type
Callable[[UsageKey, dict[str, dict[int, list[Any]]]], tuple[TaggedContent, list[Any]]]
]
if isinstance(content_key, CourseKey):
tagged_content, children = _get_course_tagged_object_and_children(
content_key, object_tag_cache
)
get_tagged_children = _get_xblock_tagged_object_and_children
elif isinstance(content_key, LibraryLocatorV2):
tagged_content, children = _get_library_tagged_object_and_children(
content_key, object_tag_cache
)
get_tagged_children = _get_library_block_tagged_object
else:
raise ValueError(f"Invalid content_key: {type(content_key)} -> {content_key}")

blocks: list[tuple[TaggedContent, list | None]] = [(tagged_content, children)]

while blocks:
tagged_block, xblock = blocks.pop()
tagged_block, block_children = blocks.pop()
tagged_block.children = []

if xblock.has_children:
for child_id in xblock.children:
child_block = store.get_item(child_id)
tagged_child = TaggedContent(
display_name=child_block.display_name_with_default,
block_id=str(child_id),
category=child_block.category,
object_tags=object_tag_cache.get(str(child_id), {}),
children=None,
)
tagged_block.children.append(tagged_child)

blocks.append((tagged_child, child_block))

return tagged_course
if not block_children:
continue

for child in block_children:
tagged_child, child_children = get_tagged_children(child, object_tag_cache)
tagged_block.children.append(tagged_child)
blocks.append((tagged_child, child_children))

return tagged_content
Loading
Loading