diff --git a/DEVELOPERS.md b/DEVELOPERS.md index 3a3a02c8a..7e4a6aa4d 100644 --- a/DEVELOPERS.md +++ b/DEVELOPERS.md @@ -144,7 +144,7 @@ If a new search engine backend class is to be implemented, it must closely follo utils.search.SearchEngineBase docstrings. There is a Django management command that can be used in order to test the implementation of a search backend. You can run it like: - docker-compose run --rm web python manage.py test_search_engine_backend -fsw --backend utils.search.backends.solr9pysolr.Solr9PySolrSearchEngine + docker compose run --rm web python manage.py test_search_engine_backend -fsw --backend utils.search.backends.solr9pysolr.Solr9PySolrSearchEngine Please read carefully the documentation of the management command to better understand how it works and how is it doing the testing. @@ -217,7 +217,7 @@ https://github.com/mtg/freesound-audio-analyzers. The docker compose of the main services for the external analyzers which depend on docker images having been previously built from the `freesound-audio-analyzers` repository. To build these images you simply need to checkout the code repository and run `make`. Once the images are built, Freesound can be run including the external analyzer services by of the docker compose -file by running `docker-compose --profile analyzers up` +file by running `docker compose --profile analyzers up` The new analysis pipeline uses a job queue based on Celery/RabbitMQ. RabbitMQ console can be accessed at port `5673` (e.g. `http://localhost:5673/rabbitmq-admin`) and using `guest` as both username and password. Also, accessing @@ -231,7 +231,7 @@ for Freesound async tasks other than analysis). - Make sure that there are no outstanding deprecation warnings for the version of django that we are upgrading to. - docker-compose run --rm web python -Wd manage.py test + docker compose run --rm web python -Wd manage.py test Check for warnings of the form `RemovedInDjango110Warning` (TODO: Make tests fail if a warning occurs) diff --git a/README.md b/README.md index d637b50f3..9e2ff7a93 100644 --- a/README.md +++ b/README.md @@ -65,35 +65,35 @@ Below are instructions for setting up a local Freesound installation for develop 8. Build all Docker containers. The first time you run this command can take a while as a number of Docker images need to be downloaded and things need to be installed and compiled. - docker-compose build + docker compose build 9. Download the [Freesound development database dump](https://drive.google.com/file/d/11z9s8GyYkVlmWdEsLSwUuz0AjZ8cEvGy/view?usp=share_link) (~6MB), uncompress it and place the resulting `freesound-small-dev-dump-2023-09.sql` in the `freesound-data/db_dev_dump/` directory. Then run the database container and load the data into it using the commands below. You should get permission to download this file from Freesound admins. - docker-compose up -d db - docker-compose run --rm db psql -h db -U freesound -d freesound -f freesound-data/db_dev_dump/freesound-small-dev-dump-2023-09.sql + docker compose up -d db + docker compose run --rm db psql -h db -U freesound -d freesound -f freesound-data/db_dev_dump/freesound-small-dev-dump-2023-09.sql # or if the above command does not work, try this one - docker-compose run --rm --no-TTY db psql -h db -U freesound -d freesound < freesound-data/db_dev_dump/freesound-small-dev-dump-2023-09.sql + docker compose run --rm --no-TTY db psql -h db -U freesound -d freesound < freesound-data/db_dev_dump/freesound-small-dev-dump-2023-09.sql 10. Update database by running Django migrations - docker-compose run --rm web python manage.py migrate + docker compose run --rm web python manage.py migrate 11. Create a superuser account to be able to log in to the local Freesound website and to the admin site - docker-compose run --rm web python manage.py createsuperuser + docker compose run --rm web python manage.py createsuperuser 12. Install static build dependencies - docker-compose run --rm web npm install --force + docker compose run --rm web npm install --force 13. Build static files. Note that this step will need to be re-run every time there are changes in Freesound's static code (JS, CSS and static media files). - docker-compose run --rm web npm run build - docker-compose run --rm web python manage.py collectstatic --noinput + docker compose run --rm web npm run build + docker compose run --rm web python manage.py collectstatic --noinput 14. Run services 🎉 - docker-compose up + docker compose up When running this command, the most important services that make Freesound work will be run locally. This includes the web application and database, but also the search engine, cache manager, queue manager and asynchronous workers, including audio processing. @@ -102,24 +102,24 @@ Below are instructions for setting up a local Freesound installation for develop 15. Build the search index, so you can search for sounds and forum posts # Open a new terminal window so the services started in the previous step keep running - docker-compose run --rm web python manage.py reindex_search_engine_sounds - docker-compose run --rm web python manage.py reindex_search_engine_forum + docker compose run --rm web python manage.py reindex_search_engine_sounds + docker compose run --rm web python manage.py reindex_search_engine_forum After following the steps, you'll have a functional Freesound installation up and running, with the most relevant services properly configured. You can run Django's shell plus command like this: - docker-compose run --rm web python manage.py shell_plus + docker compose run --rm web python manage.py shell_plus Because the `web` container mounts a named volume for the home folder of the user running the shell plus process, command history should be kept between container runs :) -16. (extra step) The steps above will get Freesound running, but to save resources in your local machine some non-essential services will not be started by default. If you look at the `docker-compose.yml` file, you'll see that some services are marked with the profile `analyzers` or `all`. These services include sound similarity, search results clustering and the audio analyzers. To run these services you need to explicitly tell `docker-compose` using the `--profile` (note that some services need additional configuration steps (see *Freesound analysis pipeline* section in `DEVELOPERS.md`): +16. (extra step) The steps above will get Freesound running, but to save resources in your local machine some non-essential services will not be started by default. If you look at the `docker compose.yml` file, you'll see that some services are marked with the profile `analyzers` or `all`. These services include sound similarity, search results clustering and the audio analyzers. To run these services you need to explicitly tell `docker compose` using the `--profile` (note that some services need additional configuration steps (see *Freesound analysis pipeline* section in `DEVELOPERS.md`): - docker-compose --profile analyzers up # To run all basic services + sound analyzers - docker-compose --profile all up # To run all services + docker compose --profile analyzers up # To run all basic services + sound analyzers + docker compose --profile all up # To run all services ### Running tests You can run tests using the Django test runner in the `web` container like that: - docker-compose run --rm web python manage.py test --settings=freesound.test_settings + docker compose run --rm web python manage.py test --settings=freesound.test_settings diff --git a/_docs/api/source/resources.rst b/_docs/api/source/resources.rst index bb96e3841..6c5c1259c 100644 --- a/_docs/api/source/resources.rst +++ b/_docs/api/source/resources.rst @@ -80,7 +80,7 @@ Filter name Type Description ``avg_rating`` numerical Average rating for the sound in the range [0, 5]. ``num_ratings`` integer Number of times the sound has been rated. ``comment`` string Textual content of the comments of a sound (tokenized). The filter is satisfied if sound contains the filter value in at least one of its comments. -``comments`` integer Number of times the sound has been commented. +``num_comments`` integer Number of times the sound has been commented. ====================== ============= ==================================================== diff --git a/accounts/tests/test_views.py b/accounts/tests/test_views.py index e29b3e3ed..2c6cd9e6a 100644 --- a/accounts/tests/test_views.py +++ b/accounts/tests/test_views.py @@ -287,14 +287,9 @@ def test_sound_search_response(self): resp = self.client.get(reverse('sounds-search')) self.assertEqual(resp.status_code, 200) - def test_geotags_box_response(self): - # 200 response on geotag box page access - resp = self.client.get(reverse('geotags-box')) - self.assertEqual(resp.status_code, 200) - - def test_geotags_box_iframe_response(self): + def test_geotags_embed_response(self): # 200 response on geotag box iframe - resp = self.client.get(reverse('embed-geotags-box-iframe')) + resp = self.client.get(reverse('embed-geotags')) self.assertEqual(resp.status_code, 200) def test_accounts_manage_pages(self): diff --git a/clustering/interface.py b/clustering/interface.py index 064b8d905..ccbbca602 100644 --- a/clustering/interface.py +++ b/clustering/interface.py @@ -109,5 +109,32 @@ def cluster_sound_results(request, features=DEFAULT_FEATURES): return {'finished': False, 'error': False} +def get_ids_in_cluster(request, requested_cluster_id): + """Get the sound ids in the requested cluster. Used for applying a filter by id when using a cluster facet. + """ + try: + requested_cluster_id = int(requested_cluster_id) - 1 + + # results are cached in clustering_utilities, available features are defined in the clustering settings file. + result = cluster_sound_results(request, features=DEFAULT_FEATURES) + results = result['result'] + + sounds_from_requested_cluster = results[int(requested_cluster_id)] + + except ValueError: + return [] + except IndexError: + return [] + except KeyError: + # If the clustering is not in cache the 'result' key won't exist + # This means that the clustering computation will be triggered asynchronously. + # Moreover, the applied clustering filter will have no effect. + # Somehow, we should inform the user that the clustering results were not available yet, and that + # he should try again later to use a clustering facet. + return [] + + return sounds_from_requested_cluster + + def hash_cache_key(key): return create_hash(key, limit=32) diff --git a/freesound.code-workspace b/freesound.code-workspace index b159aadd8..4dc94ba62 100644 --- a/freesound.code-workspace +++ b/freesound.code-workspace @@ -34,28 +34,23 @@ "tasks": { "version": "2.0.0", "tasks": [ - { - "label": "Run web and search", - "type": "shell", - "command": "docker-compose up web search", - "problemMatcher": [] - }, + { "label": "Docker compose build", "type": "shell", - "command": "docker-compose build", + "command": "docker compose build", "problemMatcher": [] }, { "label": "Build static", "type": "shell", - "command": "docker-compose run --rm web npm run build && docker-compose run --rm web python manage.py collectstatic --clear --noinput", + "command": "docker compose run --rm web npm run build && docker compose run --rm web python manage.py collectstatic --clear --noinput", "problemMatcher": [] }, { "label": "Install static", "type": "shell", - "command": "docker-compose run --rm web npm install --force", + "command": "docker compose run --rm web npm install --force", "problemMatcher": [] }, { @@ -67,37 +62,55 @@ { "label": "Create caches", "type": "shell", - "command": "docker-compose run --rm web python manage.py create_front_page_caches && docker-compose run --rm web python manage.py create_random_sounds && docker-compose run --rm web python manage.py generate_geotags_bytearray", + "command": "docker compose run --rm web python manage.py create_front_page_caches && docker compose run --rm web python manage.py create_random_sounds && docker compose run --rm web python manage.py generate_geotags_bytearray", "problemMatcher": [] }, { "label": "Run tests", "type": "shell", - "command": "docker-compose run --rm web python manage.py test --settings=freesound.test_settings", + "command": "docker compose run --rm web python manage.py test --settings=freesound.test_settings", "problemMatcher": [] }, { "label": "Run tests verbose with warnings", "type": "shell", - "command": "docker-compose run --rm web python -Wa manage.py test -v3 --settings=freesound.test_settings", + "command": "docker compose run --rm web python -Wa manage.py test -v3 --settings=freesound.test_settings", "problemMatcher": [] }, { "label": "Migrate", "type": "shell", - "command": "docker-compose run --rm web python manage.py migrate", + "command": "docker compose run --rm web python manage.py migrate", "problemMatcher": [] }, { "label": "Make migrations", "type": "shell", - "command": "docker-compose run --rm web python manage.py makemigrations", + "command": "docker compose run --rm web python manage.py makemigrations", "problemMatcher": [] }, { "label": "Shell plus", "type": "shell", - "command": "docker-compose run --rm web python manage.py shell_plus", + "command": "docker compose run --rm web python manage.py shell_plus", + "problemMatcher": [] + }, + { + "label": "Reindex search engine", + "type": "shell", + "command": "docker compose run --rm web python manage.py reindex_search_engine_sounds && docker compose run --rm web python manage.py reindex_search_engine_forum", + "problemMatcher": [] + }, + { + "label": "Post dirty sounds to search engine", + "type": "shell", + "command": "docker compose run --rm web python manage.py post_dirty_sounds_to_search_engine", + "problemMatcher": [] + }, + { + "label": "Orchestrate analysis", + "type": "shell", + "command": "docker compose run --rm web python manage.py orchestrate_analysis", "problemMatcher": [] } ] diff --git a/freesound/settings.py b/freesound/settings.py index 2223ddd38..668c2a1e9 100644 --- a/freesound/settings.py +++ b/freesound/settings.py @@ -638,6 +638,23 @@ SOLR5_BASE_URL = "http://search:8983/solr" SOLR9_BASE_URL = "http://search:8983/solr" +SEARCH_ENGINE_SIMILARITY_ANALYZERS = { + FSDSINET_ANALYZER_NAME: { + 'vector_property_name': 'embeddings', + 'vector_size': 100, + }, + FREESOUND_ESSENTIA_EXTRACTOR_NAME: { + 'vector_property_name': 'sim_vector', + 'vector_size': 100, + } +} +SEARCH_ENGINE_DEFAULT_SIMILARITY_ANALYZER = FREESOUND_ESSENTIA_EXTRACTOR_NAME +SEARCH_ENGINE_NUM_SIMILAR_SOUNDS_PER_QUERY = 500 +USE_SEARCH_ENGINE_SIMILARITY = False # Does not currently apply to API + +SEARCH_ALLOW_DISPLAY_RESULTS_IN_MAP = True +MAX_SEARCH_RESULTS_IN_MAP_DISPLAY = 10000 # This is the maximum number of sounds that will be shown when using "display results in map" mode + # ------------------------------------------------------------------------------- # Similarity client settings SIMILARITY_ADDRESS = 'similarity' diff --git a/freesound/static/bw-frontend/src/components/mapsMapbox.js b/freesound/static/bw-frontend/src/components/mapsMapbox.js index e72a97597..98c6ab476 100644 --- a/freesound/static/bw-frontend/src/components/mapsMapbox.js +++ b/freesound/static/bw-frontend/src/components/mapsMapbox.js @@ -292,7 +292,7 @@ function makeSoundsMap(geotags_url, map_element_id, on_built_callback, on_bounds if (nSounds > 1){ // The padding and offset "manual" adjustments of bounds below are to make the boudns more similar to // those created in the mapbox static maps - map.fitBounds(bounds, {duration:0, offset:[-10, 0], padding: {top:60, right:60, left:0, bottom:50}}); + map.fitBounds(bounds, {duration:0, offset:[0, 0], padding: {top:60, right:60, left:60, bottom:60}}); } else { map.setZoom(3); if (nSounds > 0){ diff --git a/freesound/static/bw-frontend/src/pages/map.js b/freesound/static/bw-frontend/src/pages/map.js index 4c1a512e4..8c8c1182e 100644 --- a/freesound/static/bw-frontend/src/pages/map.js +++ b/freesound/static/bw-frontend/src/pages/map.js @@ -12,10 +12,6 @@ const tagFilterInput = document.getElementById("tagFilter"); let currentLat; let currentLon; let currentZoom; -let currentBoxBlLa; -let currentBoxBlLon; -let currentBoxTrLat; -let currentBoxTrLon; const toggleEmbedControls = () => { if (embedControls.classList.contains('display-none')){ @@ -42,22 +38,21 @@ const updateQueryStringParameter = (uri, key, value) => { } } -const updateEmbedCode = (mapElementId, lat, lon, zoom, boxBlLat, boxBlLon, boxTrLat, boxTrLon) => { +const updateEmbedCode = (mapElementId, lat, lon, zoom) => { if (embedCodeElement === null){ return; } - - const mapCanvas = document.getElementById(mapElementId); + let mapCanvas; + if (mapElementId === undefined){ + mapCanvas = document.getElementsByClassName('main-map')[0]; + } else { + mapCanvas = document.getElementById(mapElementId); + } // Store lat, lon and zoom globally so we can use them later to call updateEmbedCode without accessing map currentLat = lat; currentLon = lon; currentZoom = zoom; - currentBoxBlLa = boxBlLat; - currentBoxBlLon = boxBlLon; - currentBoxTrLat = boxTrLat; - currentBoxTrLon = boxTrLon; // Generate embed code - const box = "#box=" + boxBlLat + "," + boxBlLon+"," + boxTrLat+"," + boxTrLon; const width = parseInt(embedWidthInputElement.value, 10); const height = parseInt(embedHeightInputElement.value, 10); let cluster = 'on'; @@ -66,16 +61,19 @@ const updateEmbedCode = (mapElementId, lat, lon, zoom, boxBlLat, boxBlLon, boxTr } let embedCode = ""; + if (mapCanvas.dataset.mapQp !== ""){ + embedCode += "&qp=" + mapCanvas.dataset.mapQp; + } + embedCode += "\" width=\"" + width + "\" height=\"" + height + "\">"; embedCodeElement.innerText = embedCode; // Update page URL so it can directly be used to share the map @@ -87,7 +85,7 @@ const updateEmbedCode = (mapElementId, lat, lon, zoom, boxBlLat, boxBlLon, boxTr } const changeEmbedWidthHeightCluster = () => { - updateEmbedCode(undefined, currentLat, currentLon, currentZoom, currentBoxBlLa, currentBoxBlLon, currentBoxTrLat, currentBoxTrLon); + updateEmbedCode(undefined, currentLat, currentLon, currentZoom); } const initMap = (mapCanvas) => { @@ -107,7 +105,7 @@ const initMap = (mapCanvas) => { [embedWidthInputElement, embedHeightInputElement, embedClusterCheckElement].forEach(element => { if (element !== null){ element.addEventListener('change', () => { - changeEmbedWidthHeightCluster(); + changeEmbedWidthHeightCluster(); }); } }); @@ -140,13 +138,6 @@ const initMap = (mapCanvas) => { zoom = mapCanvas.dataset.mapZoom; } let url = mapCanvas.dataset.geotagsUrl; - const urlBox = mapCanvas.dataset.geotagsUrlBox; - const box = document.location.hash.slice(5, document.location.hash.length); - if (box !== ''){ - // If box is given, get the geotags only from that box - url = `${urlBox}?box=${box}`; - } - const showSearch = (mapCanvas.dataset.mapShowSearch !== undefined && mapCanvas.dataset.mapShowSearch === 'true'); const showStyleSelector = true; const clusterGeotags = true; @@ -156,7 +147,11 @@ const initMap = (mapCanvas) => { if (loadingIndicator !== null){ loadingIndicator.innerText = `${numLoadedSounds} sound${ numLoadedSounds === 1 ? '': 's'}`; } + embedWidthInputElement.value = mapCanvas.offsetWidth; + embedHeightInputElement.value = mapCanvas.offsetHeight; }, updateEmbedCode, centerLat, centerLon, zoom, showSearch, showStyleSelector, clusterGeotags, showMapEvenIfNoGeotags); + + } export { initMap }; \ No newline at end of file diff --git a/freesound/static/bw-frontend/src/pages/search.js b/freesound/static/bw-frontend/src/pages/search.js index 2b9c13428..6f73a7178 100644 --- a/freesound/static/bw-frontend/src/pages/search.js +++ b/freesound/static/bw-frontend/src/pages/search.js @@ -96,7 +96,8 @@ var filter_in_remix_group_element = document.getElementById('filter_in_remix_gro var sort_by_element = document.getElementById('sort-by'); var group_by_pack_element = document.getElementById('group_by_pack'); var only_sounds_with_pack_element = document.getElementById('only_sounds_with_pack'); -var use_compact_mode_element = document.getElementById('use_compact_mode'); +var use_compact_mode_element = document.getElementById('use_compact_mode'); +var use_map_mode_element = document.getElementById('use_map_mode'); function update_hidden_compact_mode_element() { var hiddenElement = document.getElementById('use_compact_mode_hidden'); @@ -112,6 +113,20 @@ use_compact_mode_element.addEventListener('change', function() { update_hidden_compact_mode_element() }) +function update_hidden_map_mode_element() { + var hiddenElement = document.getElementById('use_map_mode_hidden'); + if (use_map_mode_element.checked) { + hiddenElement.value = "1"; + } else { + hiddenElement.value = "0"; + } +} + +update_hidden_map_mode_element() +use_map_mode_element.addEventListener('change', function() { + update_hidden_map_mode_element() +}) + function advancedSearchOptionsIsVisible() { return advanced_search_hidden_field.value === "1"; diff --git a/freesound/test_settings.py b/freesound/test_settings.py index d95458f5a..3308d0030 100644 --- a/freesound/test_settings.py +++ b/freesound/test_settings.py @@ -14,6 +14,7 @@ '--with-xunit', ] +AKISMET_KEY = '' # Avoid making requests to "real" Akismet server if running SECRET_KEY = "testsecretwhichhastobeatleast16characterslong" SUPPORT = (('Name Surname', 'support@freesound.org'),) STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' diff --git a/freesound/urls.py b/freesound/urls.py index 532294fae..0a87522af 100644 --- a/freesound/urls.py +++ b/freesound/urls.py @@ -82,7 +82,7 @@ path('charts/', accounts.views.charts, name="charts"), path('embed/sound/iframe//simple//', sounds.views.embed_iframe, name="embed-simple-sound-iframe"), - path('embed/geotags_box/iframe/', geotags.views.embed_iframe, name="embed-geotags-box-iframe"), + path('embed/geotags_box/iframe/', geotags.views.embed_iframe, name="embed-geotags"), path('oembed/', sounds.views.oembed, name="oembed-sound"), path('after-download-modal/', sounds.views.after_download_modal, name="after-download-modal"), @@ -93,7 +93,7 @@ path('browse/packs/', sounds.views.packs, name="packs"), path('browse/random/', sounds.views.random, name="sounds-random"), re_path(r'^browse/geotags/(?P[\w-]+)?/?$', geotags.views.geotags, name="geotags"), - path('browse/geotags_box/', geotags.views.geotags_box, name="geotags-box"), + path('browse/query/', geotags.views.for_query, name="geotags-query"), path('contact/', support.views.contact, name="contact"), diff --git a/general/tasks.py b/general/tasks.py index 2dea3c913..d7e0209d6 100644 --- a/general/tasks.py +++ b/general/tasks.py @@ -260,10 +260,12 @@ def process_analysis_results(sound_id, analyzer, status, analysis_time, exceptio {'task_name': PROCESS_ANALYSIS_RESULTS_TASK_NAME, 'sound_id': sound_id, 'analyzer': analyzer, 'status': status, 'exception': str(exception), 'work_time': round(time.time() - start_time)})) else: - # Load analysis output to database field (following configuration in settings.ANALYZERS_CONFIGURATION) + # Load analysis output to database field (following configuration in settings.ANALYZERS_CONFIGURATION) a.load_analysis_data_from_file_to_db() - # Set sound to index dirty so that the sound gets reindexed with updated analysis fields - a.sound.mark_index_dirty(commit=True) + + if analyzer in settings.SEARCH_ENGINE_SIMILARITY_ANALYZERS or analyzer in settings.ANALYZERS_CONFIGURATION: + # If the analyzer produces data that should be indexed in the search engine, set sound index to dirty so that the sound gets reindexed soon + a.sound.mark_index_dirty(commit=True) workers_logger.info("Finished processing analysis results (%s)" % json.dumps( {'task_name': PROCESS_ANALYSIS_RESULTS_TASK_NAME, 'sound_id': sound_id, 'analyzer': analyzer, 'status': status, 'work_time': round(time.time() - start_time)})) diff --git a/geotags/tests.py b/geotags/tests.py index 53c0fb7c4..30778961a 100644 --- a/geotags/tests.py +++ b/geotags/tests.py @@ -40,13 +40,8 @@ def test_browse_geotags(self): check_values = {'tag': 'soundscape', 'username': None} self.check_context(resp.context, check_values) - def test_browse_geotags_box(self): - resp = self.client.get(reverse('geotags-box')) - check_values = {'center_lat': None, 'center_lon': None, 'zoom': None, 'username': None} - self.check_context(resp.context, check_values) - - def test_geotags_box_iframe(self): - resp = self.client.get(reverse('embed-geotags-box-iframe')) + def test_geotags_embed(self): + resp = self.client.get(reverse('embed-geotags')) check_values = {'m_width': 942, 'm_height': 600, 'cluster': True, 'center_lat': None, 'center_lon': None, 'zoom': None, 'username': None} self.check_context(resp.context, check_values) @@ -96,3 +91,8 @@ def test_browse_geotags_case_insensitive(self): # Response contains 3 int32 objects per sound: id, lat and lng. Total size = 3 * 4 bytes = 12 bytes n_sounds = len(resp.content) // 12 self.assertEqual(n_sounds, 2) + + def test_browse_geotags_for_query(self): + resp = self.client.get(reverse('geotags-query') + f'?q=barcelona') + check_values = {'query_description': 'barcelona'} + self.check_context(resp.context, check_values) diff --git a/geotags/urls.py b/geotags/urls.py index 2778b48a8..be31973b3 100644 --- a/geotags/urls.py +++ b/geotags/urls.py @@ -26,7 +26,7 @@ path('sounds_barray/user_latest//', geotags.geotags_for_user_latest_barray, name="geotags-for-user-latest-barray"), path('sounds_barray/pack//', geotags.geotags_for_pack_barray, name="geotags-for-pack-barray"), path('sounds_barray/sound//', geotags.geotag_for_sound_barray, name="geotags-for-sound-barray"), + path('sounds_barray/query/', geotags.geotags_for_query_barray, name="geotags-for-query-barray"), re_path(r'^sounds_barray/(?P[\w-]+)?/?$', geotags.geotags_barray, name="geotags-barray"), - path('geotags_box_barray/', geotags.geotags_box_barray, name="geotags-box-barray"), path('infowindow//', geotags.infowindow, name="geotags-infowindow"), ] diff --git a/geotags/views.py b/geotags/views.py index 4df4ab535..b7ca034ba 100644 --- a/geotags/views.py +++ b/geotags/views.py @@ -23,6 +23,7 @@ import logging import math import struct +import urllib.parse from django.conf import settings from django.core.cache import cache @@ -31,9 +32,12 @@ from django.urls import reverse from django.views.decorators.cache import cache_page from django.views.decorators.clickjacking import xframe_options_exempt +from accounts.models import Profile +from search.views import search_prepare_parameters from sounds.models import Sound, Pack from utils.logging_filters import get_client_ip +from utils.search.search_sounds import perform_search_engine_query from utils.username import redirect_if_old_username_or_404, raise_404_if_user_is_deleted web_logger = logging.getLogger('web') @@ -44,17 +48,50 @@ def log_map_load(map_type, num_geotags, request): 'map_type': map_type, 'num_geotags': num_geotags, 'ip': get_client_ip(request)})) -def generate_bytearray(sound_queryset): +def update_query_params_for_map_query(query_params, preserve_facets=False): + # Force is_geotagged filter to be present + if query_params['query_filter']: + if 'is_geotagged' not in query_params['query_filter']: + query_params['query_filter'] = query_params['query_filter'] + ' is_geotagged:1' + else: + query_params['query_filter'] = 'is_geotagged:1' + # Force one single page with "all" results, and don't group by pack + query_params.update({ + 'current_page': 1, + 'num_sounds': settings.MAX_SEARCH_RESULTS_IN_MAP_DISPLAY, + 'group_by_pack': False, + 'only_sounds_with_pack': False, + 'field_list': ['id', 'score', 'geotag'] + }) + if not preserve_facets: + # No need to compute facets for the bytearray, but it might be needed for the main query + if 'facets' in query_params: + del query_params['facets'] + + +def generate_bytearray(sound_queryset_or_list): # sounds as bytearray packed_sounds = io.BytesIO() num_sounds_in_bytearray = 0 - for s in sound_queryset: - if not math.isnan(s.geotag.lat) and not math.isnan(s.geotag.lon): - packed_sounds.write(struct.pack("i", s.id)) - packed_sounds.write(struct.pack("i", int(s.geotag.lat*1000000))) - packed_sounds.write(struct.pack("i", int(s.geotag.lon*1000000))) - num_sounds_in_bytearray += 1 - + for s in sound_queryset_or_list: + if type(s) == Sound: + if not math.isnan(s.geotag.lat) and not math.isnan(s.geotag.lon): + packed_sounds.write(struct.pack("i", s.id)) + packed_sounds.write(struct.pack("i", int(s.geotag.lat * 1000000))) + packed_sounds.write(struct.pack("i", int(s.geotag.lon * 1000000))) + num_sounds_in_bytearray += 1 + elif type(s) == dict: + try: + lon, lat = s['geotag'][0].split(' ') + lat = max(min(float(lat), 90), -90) + lon = max(min(float(lon), 180), -180) + packed_sounds.write(struct.pack("i", s['id'])) + packed_sounds.write(struct.pack("i", int(lat * 1000000))) + packed_sounds.write(struct.pack("i", int(lon * 1000000))) + num_sounds_in_bytearray += 1 + except: + pass + return packed_sounds.getvalue(), num_sounds_in_bytearray @@ -77,37 +114,18 @@ def geotags_barray(request, tag=None): return HttpResponse(generated_bytearray, content_type='application/octet-stream') -def geotags_box_barray(request): - box = request.GET.get("box", "-180,-90,180,90") - is_embed = request.GET.get("embed", "0") == "1" - try: - min_lat, min_lon, max_lat, max_lon = box.split(",") - qs = Sound.objects.select_related("geotag").exclude(geotag=None).filter(moderation_state="OK", processing_state="OK") - sounds = [] - if min_lat <= max_lat and min_lon <= max_lon: - sounds = qs.filter(geotag__lat__range=(min_lat, max_lat)).filter(geotag__lon__range=(min_lon, max_lon)) - elif min_lat > max_lat and min_lon <= max_lon: - sounds = qs.exclude(geotag__lat__range=(max_lat, min_lat)).filter(geotag__lon__range=(min_lon, max_lon)) - elif min_lat <= max_lat and min_lon > max_lon: - sounds = qs.filter(geotag__lat__range=(min_lat, max_lat)).exclude(geotag__lon__range=(max_lon, min_lon)) - elif min_lat > max_lat and min_lon > max_lon: - sounds = qs.exclude(geotag__lat__range=(max_lat, min_lat)).exclude(geotag__lon__range=(max_lon, min_lon)) - - generated_bytearray, num_geotags = generate_bytearray(sounds) - if num_geotags > 0: - log_map_load('box-embed' if is_embed else 'box', num_geotags, request) - return HttpResponse(generated_bytearray, content_type='application/octet-stream') - except ValueError: - raise Http404 - - @redirect_if_old_username_or_404 @raise_404_if_user_is_deleted @cache_page(60 * 15) def geotags_for_user_barray(request, username): + profile = get_object_or_404(Profile, user__username=username) is_embed = request.GET.get("embed", "0") == "1" - sounds = Sound.public.select_related('geotag').filter(user__username__iexact=username).exclude(geotag=None) - generated_bytearray, num_geotags = generate_bytearray(sounds) + results, _ = perform_search_engine_query({ + 'query_filter': f'username:"{username}" is_geotagged:1', # No need to urlencode here as it will happpen somwhere before sending query to solr + 'field_list': ['id', 'score', 'geotag'], + 'num_sounds': profile.num_sounds, + }) + generated_bytearray, num_geotags = generate_bytearray(results.docs) if num_geotags > 0: log_map_load('user-embed' if is_embed else 'user', num_geotags, request) return HttpResponse(generated_bytearray, content_type='application/octet-stream') @@ -124,8 +142,13 @@ def geotags_for_user_latest_barray(request, username): def geotags_for_pack_barray(request, pack_id): - sounds = Sound.public.select_related('geotag').filter(pack__id=pack_id).exclude(geotag=None) - generated_bytearray, num_geotags = generate_bytearray(sounds) + pack = get_object_or_404(Pack, id=pack_id) + results, _ = perform_search_engine_query({ + 'query_filter': f'grouping_pack:"{pack.id}_{pack.name}" is_geotagged:1', # No need to urlencode here as it will happpen somwhere before sending query to solr + 'field_list': ['id', 'score', 'geotag'], + 'num_sounds': pack.num_sounds, + }) + generated_bytearray, num_geotags = generate_bytearray(results.docs) if num_geotags > 0: log_map_load('pack', num_geotags, request) return HttpResponse(generated_bytearray, content_type='application/octet-stream') @@ -139,6 +162,24 @@ def geotag_for_sound_barray(request, sound_id): return HttpResponse(generated_bytearray, content_type='application/octet-stream') +def geotags_for_query_barray(request): + results_cache_key = request.GET.get('key', None) + if results_cache_key is not None: + # If cache key is present, use it to get the results + results_docs = cache.get(results_cache_key) + else: + # Otherwise, perform a search query to get the results + query_params, _, _ = search_prepare_parameters(request) + update_query_params_for_map_query(query_params) + results, _ = perform_search_engine_query(query_params) + results_docs = results.docs + + generated_bytearray, num_geotags = generate_bytearray(results_docs) + if num_geotags > 0: + log_map_load('query', num_geotags, request) + return HttpResponse(generated_bytearray, content_type='application/octet-stream') + + def _get_geotags_query_params(request): return { 'center_lat': request.GET.get('c_lat', None), @@ -146,13 +187,15 @@ def _get_geotags_query_params(request): 'zoom': request.GET.get('z', None), 'username': request.GET.get('username', None), 'pack': request.GET.get('pack', None), - 'tag': request.GET.get('tag', None) + 'tag': request.GET.get('tag', None), + 'query_params': urllib.parse.unquote(request.GET['qp']) if 'qp' in request.GET else None # This is used for map embeds based on general queries } def geotags(request, tag=None): tvars = _get_geotags_query_params(request) if tag is None: + query_search_page_url = '' url = reverse('geotags-barray') # If "all geotags map" and no lat/lon/zoom is indicated, center map so whole world is visible if tvars['center_lat'] is None: @@ -163,12 +206,14 @@ def geotags(request, tag=None): tvars['zoom'] = 2 else: url = reverse('geotags-barray', args=[tag]) + query_search_page_url = reverse('sounds-search') + f'?f=tag:{tag}&mm=1' tvars.update({ # Overwrite tag and username query params (if present) 'tag': tag, 'username': None, 'pack': None, 'url': url, + 'query_search_page_url': query_search_page_url }) return render(request, 'geotags/geotags.html', tvars) @@ -183,6 +228,7 @@ def for_user(request, username): 'pack': None, 'sound': None, 'url': reverse('geotags-for-user-barray', args=[username]), + 'query_search_page_url': reverse('sounds-search') + f'?f=username:{username}&mm=1' }) return render(request, 'geotags/geotags.html', tvars) @@ -223,6 +269,7 @@ def for_pack(request, username, pack_id): 'pack': pack, 'sound': None, 'url': reverse('geotags-for-pack-barray', args=[pack.id]), + 'query_search_page_url': reverse('sounds-search') + f'?f=grouping_pack:"{pack.id}_{urllib.parse.quote(pack.name)}"&mm=1', 'modal_version': request.GET.get('ajax'), }) if request.GET.get('ajax'): @@ -233,12 +280,32 @@ def for_pack(request, username, pack_id): return render(request, 'geotags/geotags.html', tvars) -def geotags_box(request): - # This view works the same as "geotags" but it takes the username/tag parameter from query parameters and - # onyl gets the geotags for a specific bounding box specified via hash parameters. - # Currently we are only keeping this as legacy because it is not used anymore but there might still be - # links pointing to it. +def for_query(request): tvars = _get_geotags_query_params(request) + request_parameters_string = request.get_full_path().split('?')[-1] + q = request.GET.get('q', None) + f = request.GET.get('f', None) + query_description = '' + if q is None and f is None: + query_description = 'Empty query' + elif q is not None and f is not None: + query_description = f'{q} (some filters applied)' + else: + if q is not None: + query_description = q + if f is not None: + query_description = f'Empty query with some filtes applied' + tvars.update({ + 'tag': None, + 'username': None, + 'pack': None, + 'sound': None, + 'query_params': request_parameters_string, + 'query_params_encoded': urllib.parse.quote(request_parameters_string), + 'query_search_page_url': reverse('sounds-search') + f'?{request_parameters_string}', + 'query_description': query_description, + 'url': reverse('geotags-for-query-barray') + f'?{request_parameters_string}', + }) return render(request, 'geotags/geotags.html', tvars) @@ -251,7 +318,7 @@ def embed_iframe(request): 'cluster': request.GET.get('c', 'on') != 'off' }) tvars.update({'mapbox_access_token': settings.MAPBOX_ACCESS_TOKEN}) - return render(request, 'embeds/geotags_box_iframe.html', tvars) + return render(request, 'embeds/geotags_embed.html', tvars) def infowindow(request, sound_id): diff --git a/search/templatetags/search.py b/search/templatetags/search.py index 6f617de1c..b847f8a7a 100644 --- a/search/templatetags/search.py +++ b/search/templatetags/search.py @@ -92,6 +92,10 @@ def display_facet(context, flt, facet, facet_type, title=""): context['sort'] if context['sort'] is not None else '', context['weights'] or '' ) + if context['similar_to'] is not None: + element['add_filter_url'] += '&similar_to={}'.format(context['similar_to']) + if context['use_map_mode'] == True: + element['add_filter_url'] += '&mm=1' filtered_facet.append(element) # We sort the facets by count. Also, we apply an opacity filter on "could" type pacets diff --git a/search/tests.py b/search/tests.py index 8f76da1e0..388a55337 100644 --- a/search/tests.py +++ b/search/tests.py @@ -20,7 +20,7 @@ from django.core.cache import cache from django.test import TestCase -from django.test.utils import skipIf +from django.test.utils import skipIf, override_settings from django.urls import reverse from sounds.models import Sound from utils.search import SearchResults, SearchResultsPaginator @@ -142,6 +142,7 @@ def test_search_page_response_ok(self, perform_search_engine_query): self.assertEqual(resp.context['error_text'], None) self.assertEqual(len(resp.context['docs']), self.NUM_RESULTS) + @mock.patch('search.views.perform_search_engine_query') def test_search_page_num_queries(self, perform_search_engine_query): perform_search_engine_query.return_value = self.perform_search_engine_query_response @@ -155,16 +156,32 @@ def test_search_page_num_queries(self, perform_search_engine_query): cache.clear() with self.assertNumQueries(1): self.client.get(reverse('sounds-search') + '?cm=1') - - # Now check number of queries when displaying results as packs (i.e., searching for packs) - cache.clear() - with self.assertNumQueries(5): - self.client.get(reverse('sounds-search') + '?only_p=1') - - # Also check packs when displaying in grid mode - cache.clear() - with self.assertNumQueries(5): - self.client.get(reverse('sounds-search') + '?only_p=1&cm=1') + + with override_settings(USE_SEARCH_ENGINE_SIMILARITY=True): + # When using search engine similarity, there'll be one extra query performed to get the similarity status of the sounds + + # Now check number of queries when displaying results as packs (i.e., searching for packs) + cache.clear() + with self.assertNumQueries(6): + self.client.get(reverse('sounds-search') + '?only_p=1') + + # Also check packs when displaying in grid mode + cache.clear() + with self.assertNumQueries(6): + self.client.get(reverse('sounds-search') + '?only_p=1&cm=1') + + with override_settings(USE_SEARCH_ENGINE_SIMILARITY=False): + # When not using search engine similarity, there'll be one less query performed as similarity state is retrieved directly from sound object + + # Now check number of queries when displaying results as packs (i.e., searching for packs) + cache.clear() + with self.assertNumQueries(5): + self.client.get(reverse('sounds-search') + '?only_p=1') + + # Also check packs when displaying in grid mode + cache.clear() + with self.assertNumQueries(5): + self.client.get(reverse('sounds-search') + '?only_p=1&cm=1') @mock.patch('search.views.perform_search_engine_query') def test_search_page_with_filters(self, perform_search_engine_query): diff --git a/search/views.py b/search/views.py index 79ea71928..2ea1cfea6 100644 --- a/search/views.py +++ b/search/views.py @@ -22,6 +22,7 @@ import json import logging import re +import uuid import sentry_sdk from collections import defaultdict, Counter @@ -33,10 +34,12 @@ import forum import sounds +import geotags from clustering.clustering_settings import DEFAULT_FEATURES, NUM_SOUND_EXAMPLES_PER_CLUSTER_FACET, \ NUM_TAGS_SHOWN_PER_CLUSTER_FACET from clustering.interface import cluster_sound_results, get_sound_ids_from_search_engine_query from forum.models import Post +from utils.encryption import create_hash from utils.logging_filters import get_client_ip from utils.ratelimit import key_for_ratelimiting, rate_per_ip from utils.search.search_sounds import perform_search_engine_query, search_prepare_parameters, \ @@ -49,28 +52,17 @@ def search_view_helper(request, tags_mode=False): query_params, advanced_search_params_dict, extra_vars = search_prepare_parameters(request) - # check if there was a filter parsing error + # Check if there was a filter parsing error if extra_vars['parsing_error']: search_logger.info(f"Query filter parsing error. filter: {request.GET.get('f', '')}") extra_vars.update({'error_text': 'There was an error while searching, is your query correct?'}) return extra_vars - # get the url query params for later sending it to the clustering engine + # Get the url query params for later sending it to the clustering engine (this is only used with the clustering feature) url_query_params_string = request.META['QUERY_STRING'] - # get sound ids of the requested cluster when applying a clustering facet - # the list of ids is used later on to create a Solr query with filter by ids in - cluster_id = request.GET.get('cluster_id') - - if settings.ENABLE_SEARCH_RESULTS_CLUSTERING and cluster_id: - in_ids = _get_ids_in_cluster(request, cluster_id) - else: - in_ids = [] - query_params.update({'only_sounds_within_ids': in_ids}) - - query_params.update({'facets': settings.SEARCH_SOUNDS_DEFAULT_FACETS}) - - filter_query_split = split_filter_query(query_params['query_filter'], extra_vars['parsed_filters'], cluster_id) + # Get a "split" version of the filter which is used to display filters in UI and for some other checks (see below) + filter_query_split = split_filter_query(query_params['query_filter'], extra_vars['parsed_filters'], extra_vars['cluster_id']) # Get tags taht are being used in filters (this is used later to remove them from the facet and also for tags mode) tags_in_filter = [] @@ -85,7 +77,6 @@ def search_view_helper(request, tags_mode=False): # Process tags mode stuff initial_tagcloud = None if tags_mode: - # In tags mode, we increase the size of the tags facet so we include more related tags query_params['facets'][settings.SEARCH_SOUNDS_FIELD_TAGS]['limit'] = 50 @@ -110,7 +101,6 @@ def search_view_helper(request, tags_mode=False): 'initial_tagcloud': initial_tagcloud, } - # In the tvars section we pass the original group_by_pack value to avoid it being set to false if there is a pack filter (see search_prepare_parameters) # This is so that we keep track of the original setting of group_by_pack before the filter was applied, and so that if the pack filter is removed, we can # automatically revert to the previous group_by_pack setting. Also, we compute "disable_group_by_pack_option" so that when we have changed the real @@ -122,21 +112,42 @@ def search_view_helper(request, tags_mode=False): disable_only_sounds_by_pack_option= 'pack:' in query_params['query_filter'] only_sounds_with_pack = "1" if query_params['only_sounds_with_pack'] else "" if only_sounds_with_pack: - # If displaying seachr results as packs, include 3 sounds per pack group in the results so we can display these sounds as selected sounds in the + # If displaying search results as packs, include 3 sounds per pack group in the results so we can display these sounds as selected sounds in the # display_pack templatetag query_params['num_sounds_per_pack_group'] = 3 + # Parpare variables for map view + disable_display_results_in_grid_option = False + map_bytearray_url = '' + use_map_mode = settings.SEARCH_ALLOW_DISPLAY_RESULTS_IN_MAP and request.GET.get("mm", "0") == "1" + map_mode_query_results_cache_key = None + open_in_map_url = None + if use_map_mode: + # Prepare some URLs for loading sounds and providing links to map + current_query_params = request.get_full_path().split("?")[-1] + open_in_map_url = reverse('geotags-query') + f'?{current_query_params}' + map_mode_query_results_cache_key = f'map-query-results-{create_hash(current_query_params, 10)}' + map_bytearray_url = reverse('geotags-for-query-barray') + f'?key={map_mode_query_results_cache_key}' + # Update some query parameters and options to adapt to map mode + disable_group_by_pack_option = True + disable_only_sounds_by_pack_option = True + disable_display_results_in_grid_option = True + geotags.views.update_query_params_for_map_query(query_params, preserve_facets=True) + + tvars = { 'error_text': None, 'filter_query': query_params['query_filter'], 'filter_query_split': filter_query_split, 'search_query': query_params['textual_query'], + 'similar_to': query_params['similar_to'], 'group_by_pack_in_request': "1" if group_by_pack_in_request else "", 'disable_group_by_pack_option': disable_group_by_pack_option, 'only_sounds_with_pack': only_sounds_with_pack, 'only_sounds_with_pack_in_request': "1" if only_sounds_with_pack_in_request else "", 'disable_only_sounds_by_pack_option': disable_only_sounds_by_pack_option, 'use_compact_mode': should_use_compact_mode(request), + 'disable_display_results_in_grid_option': disable_display_results_in_grid_option, 'advanced': extra_vars['advanced'], 'sort': query_params['sort'], 'sort_options': [(option, option) for option in settings.SEARCH_SOUNDS_SORT_OPTIONS_WEB], @@ -150,44 +161,58 @@ def search_view_helper(request, tags_mode=False): 'tags_mode': tags_mode, 'tags_in_filter': tags_in_filter, 'has_advanced_search_settings_set': contains_active_advanced_search_filters(request, query_params, extra_vars), - 'advanced_search_closed_on_load': settings.ADVANCED_SEARCH_MENU_ALWAYS_CLOSED_ON_PAGE_LOAD + 'advanced_search_closed_on_load': settings.ADVANCED_SEARCH_MENU_ALWAYS_CLOSED_ON_PAGE_LOAD, + 'allow_map_mode': settings.SEARCH_ALLOW_DISPLAY_RESULTS_IN_MAP, + 'use_map_mode': use_map_mode, + 'map_bytearray_url': map_bytearray_url, + 'open_in_map_url': open_in_map_url, + 'max_search_results_map_mode': settings.MAX_SEARCH_RESULTS_IN_MAP_DISPLAY } - tvars.update(advanced_search_params_dict) try: results, paginator = perform_search_engine_query(query_params) - if not only_sounds_with_pack: - resultids = [d.get("id") for d in results.docs] - resultsounds = sounds.models.Sound.objects.bulk_query_id(resultids) - allsounds = {} - for s in resultsounds: - allsounds[s.id] = s - # allsounds will contain info from all the sounds returned by bulk_query_id. This should - # be all sounds in docs, but if solr and db are not synchronised, it might happen that there - # are ids in docs which are not found in bulk_query_id. To avoid problems we remove elements - # in docs that have not been loaded in allsounds. - docs = [doc for doc in results.docs if doc["id"] in allsounds] - for d in docs: - d["sound"] = allsounds[d["id"]] + if not use_map_mode: + if not only_sounds_with_pack: + resultids = [d.get("id") for d in results.docs] + resultsounds = sounds.models.Sound.objects.bulk_query_id(resultids) + allsounds = {} + for s in resultsounds: + allsounds[s.id] = s + # allsounds will contain info from all the sounds returned by bulk_query_id. This should + # be all sounds in docs, but if solr and db are not synchronised, it might happen that there + # are ids in docs which are not found in bulk_query_id. To avoid problems we remove elements + # in docs that have not been loaded in allsounds. + docs = [doc for doc in results.docs if doc["id"] in allsounds] + for d in docs: + d["sound"] = allsounds[d["id"]] + else: + resultspackids = [] + sound_ids_for_pack_id = {} + for d in results.docs: + pack_id = int(d.get("group_name").split('_')[0]) + resultspackids.append(pack_id) + sound_ids_for_pack_id[pack_id] = [int(sound['id']) for sound in d.get('group_docs', [])] + resultpacks = sounds.models.Pack.objects.bulk_query_id(resultspackids, sound_ids_for_pack_id=sound_ids_for_pack_id) + allpacks = {} + for p in resultpacks: + allpacks[p.id] = p + # allpacks will contain info from all the packs returned by bulk_query_id. This should + # be all packs in docs, but if solr and db are not synchronised, it might happen that there + # are ids in docs which are not found in bulk_query_id. To avoid problems we remove elements + # in docs that have not been loaded in allsounds. + docs = [d for d in results.docs if int(d.get("group_name").split('_')[0]) in allpacks] + for d in docs: + d["pack"] = allpacks[int(d.get("group_name").split('_')[0])] else: - resultspackids = [] - sound_ids_for_pack_id = {} - for d in results.docs: - pack_id = int(d.get("group_name").split('_')[0]) - resultspackids.append(pack_id) - sound_ids_for_pack_id[pack_id] = [int(sound['id']) for sound in d.get('group_docs', [])] - resultpacks = sounds.models.Pack.objects.bulk_query_id(resultspackids, sound_ids_for_pack_id=sound_ids_for_pack_id) - allpacks = {} - for p in resultpacks: - allpacks[p.id] = p - # allpacks will contain info from all the packs returned by bulk_query_id. This should - # be all packs in docs, but if solr and db are not synchronised, it might happen that there - # are ids in docs which are not found in bulk_query_id. To avoid problems we remove elements - # in docs that have not been loaded in allsounds. - docs = [d for d in results.docs if int(d.get("group_name").split('_')[0]) in allpacks] - for d in docs: - d["pack"] = allpacks[int(d.get("group_name").split('_')[0])] + # In map we configure the search query to already return geotags data. Here we collect all this data + # and save it to the cache so we can collect it in the 'geotags_for_query_barray' view which prepares + # data points for the map of sounds. + cache.set(map_mode_query_results_cache_key, results.docs, 60 * 15) # cache for 5 minutes + + # Nevertheless we set docs to empty list as we won't displat anything in the search results page (the map + # will make an extra request that will load the cached data and display it in the map) + docs = [] search_logger.info('Search (%s)' % json.dumps({ 'ip': get_client_ip(request), @@ -205,7 +230,8 @@ def search_view_helper(request, tags_mode=False): # sure to remove the filters for the corresponding facet field thar are already active (so we remove # redundant information) if tags_in_filter: - results.facets['tag'] = [(tag, count) for tag, count in results.facets['tag'] if tag not in tags_in_filter] + if 'tag' in results.facets: + results.facets['tag'] = [(tag, count) for tag, count in results.facets['tag'] if tag not in tags_in_filter] tvars.update({ 'paginator': paginator, @@ -234,33 +260,6 @@ def search(request): return render(request, template, tvars) -def _get_ids_in_cluster(request, requested_cluster_id): - """Get the sound ids in the requested cluster. Used for applying a filter by id when using a cluster facet. - """ - try: - requested_cluster_id = int(requested_cluster_id) - 1 - - # results are cached in clustering_utilities, available features are defined in the clustering settings file. - result = cluster_sound_results(request, features=DEFAULT_FEATURES) - results = result['result'] - - sounds_from_requested_cluster = results[int(requested_cluster_id)] - - except ValueError: - return [] - except IndexError: - return [] - except KeyError: - # If the clustering is not in cache the 'result' key won't exist - # This means that the clustering computation will be triggered asynchronously. - # Moreover, the applied clustering filter will have no effect. - # Somehow, we should inform the user that the clustering results were not available yet, and that - # he should try again later to use a clustering facet. - return [] - - return sounds_from_requested_cluster - - def clustering_facet(request): """Triggers the computation of the clustering, returns the state of processing or the clustering facet. """ diff --git a/sounds/models.py b/sounds/models.py index ed47de7f4..817c4c6dc 100644 --- a/sounds/models.py +++ b/sounds/models.py @@ -412,9 +412,15 @@ def get_analyzers_data_left_join_sql(self): def get_analysis_state_essentia_exists_sql(self): """Returns the SQL bits to add analysis_state_essentia_exists to the returned data indicating if thers is a - SoundAnalysis objects existing for th given sound_id for the essentia analyzer and with status OK""" + SoundAnalysis objects existing for the given sound_id for the essentia analyzer and with status OK""" return f" exists(select 1 from sounds_soundanalysis where sounds_soundanalysis.sound_id = sound.id AND sounds_soundanalysis.analyzer = '{settings.FREESOUND_ESSENTIA_EXTRACTOR_NAME}' AND sounds_soundanalysis.analysis_status = 'OK') as analysis_state_essentia_exists," + def get_search_engine_similarity_state_sql(self): + """Returns the SQL bits to add search_engine_similarity_state to the returned data indicating if thers is a + SoundAnalysis object existing for the default similarity analyzer (settings.SEARCH_ENGINE_DEFAULT_SIMILARITY_ANALYZER) + given sound_id and with status OK""" + return f" exists(select 1 from sounds_soundanalysis where sounds_soundanalysis.sound_id = sound.id AND sounds_soundanalysis.analyzer = '{settings.SEARCH_ENGINE_DEFAULT_SIMILARITY_ANALYZER}' AND sounds_soundanalysis.analysis_status = 'OK') as search_engine_similarity_state," + def bulk_query_solr(self, sound_ids): """For each sound, get all fields needed to index the sound in Solr. Using this custom query to avoid the need of having to do some extra queries when displaying some fields related to the sound (e.g. for tags). Using this @@ -514,6 +520,7 @@ def bulk_query(self, where, order_by, limit, args, include_analyzers_output=Fals accounts_profile.has_avatar as user_has_avatar, %s %s + %s ARRAY( SELECT tags_tag.name FROM tags_tag @@ -530,7 +537,8 @@ def bulk_query(self, where, order_by, limit, args, include_analyzers_output=Fals LEFT JOIN tickets_ticket ON tickets_ticket.sound_id = sound.id %s LEFT OUTER JOIN sounds_remixgroup_sounds ON sounds_remixgroup_sounds.sound_id = sound.id - WHERE %s """ % (self.get_analysis_state_essentia_exists_sql(), + WHERE %s """ % (self.get_search_engine_similarity_state_sql(), + self.get_analysis_state_essentia_exists_sql(), self.get_analyzers_data_select_sql() if include_analyzers_output else '', ContentType.objects.get_for_model(Sound).id, self.get_analyzers_data_left_join_sql() if include_analyzers_output else '', @@ -1350,6 +1358,20 @@ def get_geotag_name(self): return f'{self.geotag_lat:.2f}, {self.geotag_lon:.3f}' else: return f'{self.geotag.lat:.2f}, {self.geotag.lon:.3f}' + + @property + def ready_for_similarity(self): + # Retruns True is the sound has been analyzed for similarity and should be available for simialrity queries + if settings.USE_SEARCH_ENGINE_SIMILARITY: + if hasattr(self, 'search_engine_similarity_state'): + # If attribute is precomputed from query (because Sound was retrieved using bulk_query), no need to perform extra queries + return self.search_engine_similarity_state + else: + # Otherwise, check if there is a SoundAnalysis object for this sound with the correct analyzer and status + return SoundAnalysis.objects.filter(sound_id=self.id, analyzer=settings.SEARCH_ENGINE_DEFAULT_SIMILARITY_ANALYZER, analysis_status='OK').exists() + else: + # If not using search engine based similarity, then use the old similarity_state DB field + return self.similarity_state == "OK" class Meta: ordering = ("-created", ) @@ -1577,7 +1599,7 @@ def bulk_query_id(self, pack_ids, sound_ids_for_pack_id=dict(), exclude_deleted= selected_sounds_data.append({ 'id': s.id, 'username': p.user.username, # Packs have same username as sounds inside pack - 'similarity_state': s.similarity_state, + 'ready_for_similarity': s.similarity_state == "OK" if not settings.USE_SEARCH_ENGINE_SIMILARITY else None, # If using search engine similarity, this needs to be retrieved later (see below) 'duration': s.duration, 'preview_mp3': s.locations('preview.LQ.mp3.url'), 'preview_ogg': s.locations('preview.LQ.ogg.url'), @@ -1585,7 +1607,7 @@ def bulk_query_id(self, pack_ids, sound_ids_for_pack_id=dict(), exclude_deleted= 'spectral': s.locations('display.spectral_bw.L.url'), 'num_ratings': s.num_ratings, 'avg_rating': s.avg_rating - }) + }) p.num_sounds_unpublished_precomputed = p.sounds.count() - p.num_sounds p.licenses_data_precomputed = ([lid for _, lid in licenses], [lname for lname, _ in licenses]) p.pack_tags = [{'name': tag, 'count': count, 'browse_url': p.browse_pack_tag_url(tag)} @@ -1596,6 +1618,16 @@ def bulk_query_id(self, pack_ids, sound_ids_for_pack_id=dict(), exclude_deleted= p.num_ratings_precomputed = len(ratings) p.avg_rating_precomputed = sum(ratings) / len(ratings) if len(ratings) else 0.0 + if settings.USE_SEARCH_ENGINE_SIMILARITY: + # To save an individual query for each selected sound, we get the similarity state of all selected sounds per pack in one single extra query + selected_sounds_ids = [] + for p in packs: + selected_sounds_ids += [s['id'] for s in p.selected_sounds_data] + sound_ids_ready_for_similarity = SoundAnalysis.objects.filter(sound_id__in=selected_sounds_ids, analyzer=settings.SEARCH_ENGINE_DEFAULT_SIMILARITY_ANALYZER, analysis_status="OK").values_list('sound_id', flat=True) + for p in packs: + for s in p.selected_sounds_data: + s['ready_for_similarity'] = s['id'] in sound_ids_ready_for_similarity + return packs def dict_ids(self, pack_ids, exclude_deleted=True): diff --git a/sounds/templatetags/display_sound.py b/sounds/templatetags/display_sound.py index 71cf40499..4d9cdbd70 100644 --- a/sounds/templatetags/display_sound.py +++ b/sounds/templatetags/display_sound.py @@ -200,7 +200,7 @@ def display_sound_no_sound_object(context, file_data, player_size, show_bookmark 'spectral': sound.locations('display.spectral_bw.L.url'), 'id': sound.id, # Only used for sounds that do actually have a sound object so we can display bookmark/similarity buttons 'username': sound.user.username, # Only used for sounds that do actually have a sound object so we can display bookmark/similarity/remix buttons - 'similarity_state': sound.similarity_state # Only used for sounds that do actually have a sound object so we can display bookmark/similarity/remix buttons + 'ready_for_similarity': sound.ready_for_similarity # Only used for sounds that do actually have a sound object so we can display bookmark/similarity/remix buttons 'remixgroup_id': sound.remixgroup_id # Only used for sounds that do actually have a sound object so we can display bookmark/similarity/remix buttons 'num_ratings': sound.num_ratings, # Used to display rating widget in players 'avg_rating': sound.avg_rating, # Used to display rating widget in players @@ -210,7 +210,7 @@ def display_sound_no_sound_object(context, file_data, player_size, show_bookmark 'sound': { 'id': file_data.get('id', file_data['preview_mp3'].split('/')[-2]), # If no id, use a unique fake ID to avoid caching problems 'username': file_data.get('username', 'nousername'), - 'similarity_state': file_data.get('similarity_state', 'FA'), + 'ready_for_similarity': file_data.get('ready_for_similarity', False), 'duration': file_data['duration'], 'samplerate': file_data.get('samplerate', 44100), 'num_ratings': file_data.get('num_ratings', 0), @@ -236,7 +236,7 @@ def display_sound_no_sound_object(context, file_data, player_size, show_bookmark }, 'show_milliseconds': 'true' if ('big' in player_size ) else 'false', 'show_bookmark_button': show_bookmark and 'id' in file_data, - 'show_similar_sounds_button': show_similar_sounds and 'similarity_state' in file_data, + 'show_similar_sounds_button': show_similar_sounds and file_data.get('ready_for_similarity', False), 'show_remix_group_button': show_remix and 'remixgroup_id' in file_data, 'show_rate_widget': 'avg_rating' in file_data, 'player_size': player_size, diff --git a/sounds/tests/test_sound.py b/sounds/tests/test_sound.py index 889b82ca5..5b6b411fa 100644 --- a/sounds/tests/test_sound.py +++ b/sounds/tests/test_sound.py @@ -793,6 +793,7 @@ def _test_similarity_update(self, cache_keys, expected, request_func, similarity self.assertEqual(self.sound.similarity_state, 'OK') self.assertContains(request_func(user) if user is not None else request_func(), expected) + @override_settings(USE_SEARCH_ENGINE_SIMILARITY=False) def test_similarity_update_display(self): self._test_similarity_update( self._get_sound_display_cache_keys(), @@ -801,6 +802,7 @@ def test_similarity_update_display(self): user=self.user, ) + @override_settings(USE_SEARCH_ENGINE_SIMILARITY=False) def test_similarity_update_view(self): self._test_similarity_update( self._get_sound_view_footer_top_cache_keys(), diff --git a/sounds/views.py b/sounds/views.py index 34ff14fdf..0a8c67596 100644 --- a/sounds/views.py +++ b/sounds/views.py @@ -65,6 +65,7 @@ from utils.nginxsendfile import sendfile, prepare_sendfile_arguments_for_sound_download from utils.pagination import paginate from utils.ratelimit import key_for_ratelimiting, rate_per_ip +from utils.search import get_search_engine, SearchEngineException from utils.search.search_sounds import get_random_sound_id_from_search_engine, perform_search_engine_query from utils.similarity_utilities import get_similar_sounds from utils.sound_upload import create_sound, NoAudioException, AlreadyExistsException, CantMoveException, \ @@ -820,13 +821,25 @@ def similar(request, username, sound_id): sound = get_object_or_404(Sound, id=sound_id, moderation_state="OK", - processing_state="OK", - similarity_state="OK") + processing_state="OK") if sound.user.username.lower() != username.lower(): raise Http404 - similarity_results, _ = get_similar_sounds( - sound, request.GET.get('preset', None), settings.NUM_SIMILAR_SOUNDS_PER_PAGE * settings.NUM_SIMILAR_SOUNDS_PAGES) + if not settings.USE_SEARCH_ENGINE_SIMILARITY: + # Get similar sounds from similarity service (gaia) + similarity_results, _ = get_similar_sounds( + sound, request.GET.get('preset', None), settings.NUM_SIMILAR_SOUNDS_PER_PAGE * settings.NUM_SIMILAR_SOUNDS_PAGES) + else: + # Get similar sounds from solr + try: + results = get_search_engine().search_sounds(similar_to=sound.id, + similar_to_max_num_sounds=settings.NUM_SIMILAR_SOUNDS_PER_PAGE * settings.NUM_SIMILAR_SOUNDS_PAGES, + num_sounds=settings.NUM_SIMILAR_SOUNDS_PER_PAGE * settings.NUM_SIMILAR_SOUNDS_PAGES) + similarity_results = [(result['id'], result['score']) for result in results.docs] + except SearchEngineException: + # Search engine not available, return empty list + similarity_results = [] + paginator = paginate(request, [sound_id for sound_id, _ in similarity_results], settings.NUM_SIMILAR_SOUNDS_PER_PAGE) similar_sounds = Sound.objects.ordered_ids(paginator['page'].object_list) tvars = {'similar_sounds': similar_sounds, 'sound': sound} diff --git a/templates/embeds/geotags_box_iframe.html b/templates/embeds/geotags_embed.html similarity index 83% rename from templates/embeds/geotags_box_iframe.html rename to templates/embeds/geotags_embed.html index 7b65fecb2..05f8b7564 100644 --- a/templates/embeds/geotags_box_iframe.html +++ b/templates/embeds/geotags_embed.html @@ -25,24 +25,19 @@ var center_lon; var zoom; - {% if username %} + {% if query_params %} + url = '{% url "geotags-for-query-barray" %}?{{ query_params|safe }}'; + {% elif username %} url = '{% url "geotags-for-user-barray" username %}?embed=1'; {% elif pack %} url = '{% url "geotags-for-pack-barray" pack %}?embed=1'; + {% elif tag %} + url = '{% url "geotags-barray" tag %}?embed=1'; {% else %} - {% if tag %} - url = '{% url "geotags-barray" tag %}?embed=1'; - {% else %} - url = '{% url "geotags-barray" %}?embed=1'; - center_lat = 24; - center_lon = 20; - zoom = 2; - var box = document.location.hash.slice(5,document.location.hash.length); - if (box !== ''){ - // If box is given, get the geotags only from that box - url = '{% url "geotags-box-barray" %}?embed=1&box=' + box; - } - {% endif %} + url = '{% url "geotags-barray" %}?embed=1'; + center_lat = 24; + center_lon = 20; + zoom = 2; {% endif %} {% if center_lat and center_lon and zoom %} diff --git a/templates/geotags/geotags.html b/templates/geotags/geotags.html index b607b9f58..8a51edf87 100644 --- a/templates/geotags/geotags.html +++ b/templates/geotags/geotags.html @@ -11,6 +11,8 @@ Map of sounds for pack {{ pack.name }} {% elif sound %} Map for sound {{ sound.original_filename }} + {% elif query_params %} + Map for query {{ query_description}} {% else %} Map of sounds {% endif %} @@ -26,6 +28,8 @@

Map of sounds for pack {{ pack.name }} {% elif sound %} Map for sound {{ sound.original_filename }} + {% elif query_params %} + Map for query {{ query_description}} {% if not modal_version %}
-
- {% if username or sound or pack %} +
+ {% if query_search_page_url %} + See results in search page + {% endif %} + {% if username or sound or pack or query_params %} View all geotags - {% else %} -
-
- -
- -
{% endif %}
-
+
{% if not sound %} - +
Embed this map
{% endif %}
diff --git a/templates/search/search.html b/templates/search/search.html index 018b734cb..6ddd8adac 100644 --- a/templates/search/search.html +++ b/templates/search/search.html @@ -36,9 +36,10 @@

Choose a tag to start browsing
-
+
+ {% if similar_to %}{% endif %} {% comment %}This is used so that we can know from JS whether we are in tags mode or not{% endcomment %} {% bw_icon 'close' %}
@@ -75,17 +76,25 @@

enable this feature. Instead, we add here the element to toggle advanced search options. In the future we might need to redesign that. {% endcomment %} -
+
{% if has_advanced_search_settings_set %}·{% endif %}
-
- Sort by: - +
+
+ Sort by: + {% if not similar_to %} + + {% else %} + + {% endif %} +
@@ -97,13 +106,13 @@

Search in
-
+
  • @@ -111,7 +120,7 @@

  • @@ -119,7 +128,7 @@

  • @@ -131,7 +140,7 @@

  • @@ -139,7 +148,7 @@

  • @@ -147,7 +156,7 @@

  • @@ -172,9 +181,9 @@

  • -
  • -
    • + {% if allow_map_mode %} +
  • + +
    • + {% endif %}
@@ -286,12 +306,13 @@

{% if filter_query_split %}
{% for filter in filter_query_split %} - + {{ filter.name }}{% bw_icon 'close' %} {% endfor %}
{% endif %} + {% if not use_map_mode %}
{% if paginator.count > 0 %} {% if use_compact_mode %} @@ -302,13 +323,13 @@

{% display_sound_small result.sound %} {% if result.n_more_in_group and result.sound.pack_id is not None %}

- {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from pack: {{ result.sound.pack_name|truncate_string:35 }} + {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from pack: {{ result.sound.pack_name|truncate_string:35 }}

{% endif %} {% else %} {% display_pack result.pack %}

- {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from pack + {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from pack

{% endif %}

@@ -321,13 +342,13 @@

{% display_sound_middle result.sound %} {% if result.n_more_in_group and result.sound.pack_id is not None %}
- {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from same pack + {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from same pack
{% endif %} {% else %} {% display_pack_big result.pack %}
- {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from this pack + {% bw_icon 'plus' %} See {{result.n_more_in_group|add:1|bw_intcomma}} result{{ result.n_more_in_group|add:1|pluralize }} from this pack
{% endif %} {% if not forloop.last %} @@ -346,6 +367,27 @@

No results... 😟
{% bw_paginator paginator page current_page request "sound" non_grouped_number_of_results %}
+ {% else %} +
+
Loading map...
+
+
+
+ See results in full map +
+
+ {% if paginator.count < max_search_results_map_mode %} +
+

{% bw_icon 'notification' %} Note that only the first {{ max_search_results_map_mode|bw_intcomma }} search results are shown on the map

+
+ {% endif %} +
+
+ {% endif %}
{% endif %} diff --git a/templates/sounds/player.html b/templates/sounds/player.html index f658834ef..3d6b463eb 100644 --- a/templates/sounds/player.html +++ b/templates/sounds/player.html @@ -5,7 +5,7 @@ data-bookmark="{% if show_bookmark_button %}true{% else %}false{% endif %}" data-bookmark-modal-url="{% if show_bookmark_button %}{% url 'bookmarks-add-form-for-sound' sound.id %}{% endif %}" data-add-bookmark-url="{% if show_bookmark_button %}{% url 'add-bookmark' sound.id %}{% endif %}" - data-similar-sounds="{% if show_similar_sounds_button and sound.similarity_state == 'OK' %}true{% else %}false{% endif %}" + data-similar-sounds="{% if show_similar_sounds_button and sound.ready_for_similarity %}true{% else %}false{% endif %}" data-similar-sounds-modal-url="{% if show_similar_sounds_button %}{% url 'sound-similar' sound.username sound.id %}?ajax=1{% endif %}" data-remix-group="{% if show_remix_group_button and sound.remixgroup_id %}true{% else %}false{% endif %}" data-remix-group-modal-url="{% if show_remix_group_button %}{% url 'sound-remixes' sound.username sound.id %}?ajax=1{% endif %}" diff --git a/templates/sounds/sound.html b/templates/sounds/sound.html index 5e222496a..7193e9529 100644 --- a/templates/sounds/sound.html +++ b/templates/sounds/sound.html @@ -65,7 +65,7 @@