Deploy recent changes to GAP

.devcontainer/Dockerfile

@@ -56,4 +56,4 @@ RUN su ${USERNAME} -c "mix local.hex --force \
 # Add Atlos-specific dependencies
 RUN apt-get update \
   && export DEBIAN_FRONTEND=noninteractive \
-  && apt-get -y install --no-install-recommends ffmpeg imagemagick curl python3 python3-dev python3-pip python3-poetry chromium
+  && apt-get -y install --no-install-recommends ffmpeg imagemagick curl python3 python3-dev python3-pip python3-poetry chromium chromium-driver

.devcontainer/devcontainer.json

@@ -21,6 +21,7 @@
 	// Use 'forwardPorts' to make a list of ports inside the container available locally.
 	// This can be used to network with other containers or with the host. 
 	"forwardPorts": [
+		1313,
 		4000,
 		4001,
 		5432
@@ -33,6 +34,12 @@
 		"ghcr.io/devcontainers/features/docker-in-docker:2": {},
 		"ghcr.io/devcontainers/features/azure-cli:1": {},
 		"ghcr.io/devcontainers/features/terraform:1": {},
-		"ghcr.io/devcontainers/features/github-cli:1": {}
+		"ghcr.io/devcontainers/features/github-cli:1": {},
+		"ghcr.io/devcontainers/features/hugo:1": {
+			"version": "0.121.2"
+		},
+		"ghcr.io/devcontainers/features/go:1": {
+			"version": "1.19.4"
+		}
 	}
 }

docs/content/docs/Incidents/attributes.md

@@ -60,4 +60,37 @@ To customize a project’s data model:
 {{< callout type="warning" >}}
 **Removing attributes during an investigation risks data integrity.**   
 If you remove an attribute from your project's data model, that attribute—and its values—will be removed from all incidents in the project.
-{{< /callout >}}
+{{< /callout >}}
+
+## Attribute groups
+In investigations with many attributes, we recommend creating attribute groups to keep your data model easily understandable for your team.
+
+Attribute groups have several components: 
+- **Name**—Use a group's name to describe the group. A location-related attributes might fit into a "Location" group.
+- **Description**—A longer text field that provides more context on the group's purpose.
+- **Color**—Use a group's color to visually distinguish it from other attribute groups.
+
+### Create an attribute group
+To create an attribute group:
+1. Navigate to the **Projects** page. 
+2. Select the relevant project. 
+3. Click on the **Manage** page and scroll down to the **Attributes** pane. 
+4. Click **Add Group**.
+5. Add a name, description, and color for the group.
+6. Click **Save**.
+
+### Add an attribute to an attribute group
+To add an existing attribute to a group:
+1. Navigate to the **Projects** page. 
+2. Select the relevant project. 
+3. Click on the **Manage** page and scroll down to the **Attributes** pane. 
+4. Drag the attribute from its current location to its new attribute group.
+
+### Exclude attribute groups from incident creation
+In investigations with many attributes, you may want to exclude some attribute groups from the new incident form so that investigators can more quickly create incidents. To exclude attribute groups from the form:
+1. Navigate to the **Projects** page. 
+2. Select the relevant project. 
+3. Click on the **Manage** page and scroll down to the **Attributes** pane.
+4. On the relevant attribute group, click **Edit Group**.
+5. Uncheck the **Include this group in the incident creation window** box. 
+6. Click **Save**.

docs/content/docs/Incidents/source-material.md

@@ -42,6 +42,24 @@ Atlos’ archival system will attempt to archive the following artifacts:
 Note that Atlos’ automatic archival system is best-effort; Atlos will not always archive every piece of media on a webpage. It is also subject to the whims of social media platforms’ changing software and policies. We think Atlos’ automatic archival tool is a convenient system for quickly preserving sources during a journalistic investigation, but it’s not suitable for forensic or legal evidentiary purposes.
 {{< /callout >}}
 
+### Disable automatic archival
+In rare cases—for example, if your investigation has a third-party archival setup—you may want to disable Atlos' automatic archival. To do so:
+1. Navigate to the **Projects** page.
+2. Select the relevant project.
+3. Click on the **Manage** page.
+4. Under **General**, expand **Advanced Options**.
+5. Disable automatic Atlos archival.
+6. Click **Save**.
+
+### Enable Internet Archive archival
+The Internet Archive is a third-party, public archive of the web. In some, less sensitive investigations, you may want to send any link added to your project to the Internet Archive for redundant archival. This archive will be visible to anyone on the internet, so do not enable this feature if your investigation is especially sensitive. To enable Internet Archive archival:
+1. Navigate to the **Projects** page.
+2. Select the relevant project.
+3. Click on the **Manage** page.
+4. Under **General**, expand **Advanced Options**.
+5. Click **Share with the Internet Archive**
+6. Click **Save**.
+
 ## Deduplication
 Deduplication, or the process of finding and eliminating duplicates, can be extremely time-consuming. To accelerate the process, Atlos deduplicates source material in two phases:
 - **Link-level deduplication—** When an investigator pastes a link into the automatic archival system or the manual upload option, Atlos will immediately notify the investigator if they’ve added a link that is already in another incident in their project.

docs/content/docs/Technical/api.md

@@ -168,11 +168,11 @@ requests.post(
 ```
 
 ### Update an incident's attribute value
-`POST /api/v2/update/:slug/:attribute_name` updates the attribute `:attribute_name` in the incident with slug `:slug`. It has two parameters:
+`POST /api/v2/update/:slug/:attribute_identifier` updates the attribute `:attribute_identifier` in the incident with slug `:slug`. It has two parameters:
 - `value`, the new value of the attribute (required). For text or single-select attributes, `value` should be a string. For multi-select attributes, `value` should be a list of strings.
 - `message`, a string to be displayed as an explanation for the update (optional). If `message` is provided, it will be added as a comment to the incident (as part of the tracked change). 
 
-The `:attribute_name` may not be the name of the attribute displayed in the interface:
+You can find the `:attribute_identifier` in the **Access** pane of your project. Attributes' names in the Atlos interface are different from their API identifiers: 
 - Core attributes have string names (such as `description` and `status`).
 - Custom attributes are identified by a long ID. 
 

platform/Dockerfile

@@ -12,9 +12,9 @@
 #   - https://pkgs.org/ - resource for finding needed packages
 #   - Ex: hexpm/elixir:1.13.4-erlang-24.3.3-debian-bullseye-20210902-slim
 #
-ARG ELIXIR_VERSION=1.16.2
-ARG OTP_VERSION=26.2.5
-ARG DEBIAN_VERSION=bookworm-20240513
+ARG ELIXIR_VERSION=1.17.2
+ARG OTP_VERSION=27.0.1
+ARG DEBIAN_VERSION=bookworm-20240812
 
 ARG BUILDER_IMAGE="hexpm/elixir:${ELIXIR_VERSION}-erlang-${OTP_VERSION}-debian-${DEBIAN_VERSION}"
 ARG RUNNER_IMAGE="debian:${DEBIAN_VERSION}"
@@ -99,7 +99,7 @@ COPY utils utils
 ENV UTILS_DIR="/app/utils"
 
 # Install packages needed for runtime environment
-RUN apt-get update -y && apt-get install -y ffmpeg imagemagick curl python3-full dbus python3-pip python3-poetry chromium tmpreaper ca-certificates postgresql-client jq \
+RUN apt-get update -y && apt-get install -y ffmpeg imagemagick curl python3-full dbus python3-pip python3-poetry chromium chromium-driver tmpreaper ca-certificates postgresql-client jq \
   && apt-get clean && rm -f /var/lib/apt/lists/*_*
 
 USER atlos

platform/deployments/README.md

@@ -26,7 +26,7 @@ To setup continuous deployment, you'll need to configure a GitHub environment wi
 
 * `AZURE_CREDENTIALS`: The output of `az ad sp create-for-rbac --name "github-actions-<staging/prod>" --role contributor --scopes /subscriptions/<subscription_id>/resourceGroups/<resource_group_id> --json-auth`
   * Note (+ TODO): This credential is _overprovisioned_. Azure shockingly doesn't have a built-in role for "deploy to Azure Container Apps", so we have to use the `contributor` role. This is a security risk, but it's the best we can do for now without getting Azure P2 or P3. They [say](https://github.com/microsoft/azure-container-apps/issues/35#issuecomment-1675072081) that they will introduce more roles in Q4 2023.
-* `AZURE_CONTAINER_APP_NAME`: The name of the Azure Container App name
+* `AZURE_CONTAINER_APP_NAME`: The name of the Azure Container App (e.g., `ca-main-platform-staging-eastus`)
 * `AZURE_CONTAINER_APP_RESOURCE_GROUP`: The name of the Azure Container Apps resource group
 
 You'll also need to be sure that a corresponding GitHub Actions workflow file exists for the given deployment.

platform/deployments/container_app.tf

@@ -67,7 +67,7 @@ resource "azurerm_container_app" "platform" {
 
     container {
       name  = "platform"
-      image = "ghcr.io/atlosdotorg/atlos:main"
+      image = "ghcr.io/atlosdotorg/atlos:v1.2.4"
 
       cpu    = 1.0
       memory = "2Gi"

platform/lib/platform/global_search.ex

@@ -21,18 +21,20 @@ defmodule Platform.GlobalSearch do
   defmemo perform_search(query, %User{} = user) when is_binary(query), expires_in: 10000 do
     query_lower_raw = String.trim(query) |> String.downcase()
 
-    query =
+    query_cleaned =
       String.trim(query)
       |> String.downcase()
 
     query_only_alphaneumeric = String.replace(query, ~r/[^a-zA-Z0-9\s\-]/, "")
-    query = query |> String.replace(~r/\s+/, " OR ") |> String.replace(~r/[^a-zA-Z0-9\s\-]/, "")
+
+    query_websearch =
+      query_cleaned |> String.replace(~r/\s+/, " OR ") |> String.replace(~r/[^a-zA-Z0-9\s\-]/, "")
 
     media_version_query =
       from(
         mv in MediaVersion,
         where:
-          fragment("? @@ websearch_to_tsquery('simple', ?)", mv.searchable, ^query) or
+          fragment("? @@ websearch_to_tsquery('simple', ?)", mv.searchable, ^query_websearch) or
             ilike(mv.source_url, ^"%#{query_only_alphaneumeric}%") or
             fragment("LOWER(?) = ?", mv.source_url, ^query_lower_raw),
         join: m in assoc(mv, :media),
@@ -48,7 +50,11 @@ defmodule Platform.GlobalSearch do
                 fragment("LOWER(?) = ?", mv.source_url, ^query_lower_raw)
             ),
           desc:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", mv.searchable, ^query),
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              mv.searchable,
+              ^query_websearch
+            ),
           desc: mv.inserted_at
         ],
         limit: 3,
@@ -59,16 +65,21 @@ defmodule Platform.GlobalSearch do
             ilike(mv.source_url, ^"%#{query_only_alphaneumeric}%") or
               fragment("LOWER(?) = ?", mv.source_url, ^query_lower_raw),
           cd_rank:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", mv.searchable, ^query)
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              mv.searchable,
+              ^query_websearch
+            )
         }
       )
 
     media_query =
       from(
         m in Media,
         where:
-          fragment("? @@ websearch_to_tsquery('simple', ?)", m.searchable, ^query) or
+          fragment("? @@ websearch_to_tsquery('simple', ?)", m.searchable, ^query_websearch) or
             ilike(m.attr_description, ^"%#{query_only_alphaneumeric}%") or
+            ilike(m.searchable_text, ^"%#{query_cleaned}%") or
             ilike(m.slug, ^"%#{query_only_alphaneumeric}%"),
         join: p in assoc(m, :project),
         join: pm in assoc(p, :memberships),
@@ -82,7 +93,11 @@ defmodule Platform.GlobalSearch do
                 ilike(m.slug, ^"%#{query_only_alphaneumeric}%")
             ),
           desc:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", m.searchable, ^query),
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              m.searchable,
+              ^query_websearch
+            ),
           desc: m.inserted_at
         ],
         limit: 3,
@@ -93,7 +108,11 @@ defmodule Platform.GlobalSearch do
             ilike(m.attr_description, ^"%#{query_only_alphaneumeric}%") or
               ilike(m.slug, ^"%#{query_only_alphaneumeric}%"),
           cd_rank:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", m.searchable, ^query)
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              m.searchable,
+              ^query_websearch
+            )
         }
       )
 
@@ -102,7 +121,7 @@ defmodule Platform.GlobalSearch do
         u in User,
         where:
           u.username != "atlos" and
-            (fragment("? @@ websearch_to_tsquery('simple', ?)", u.searchable, ^query) or
+            (fragment("? @@ websearch_to_tsquery('simple', ?)", u.searchable, ^query_websearch) or
                ilike(u.username, ^"%#{query_only_alphaneumeric}%")),
         order_by: [
           asc:
@@ -111,23 +130,31 @@ defmodule Platform.GlobalSearch do
               ilike(u.username, ^"%#{query_only_alphaneumeric}%")
             ),
           desc:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", u.searchable, ^query),
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              u.searchable,
+              ^query_websearch
+            ),
           desc: u.inserted_at
         ],
         limit: 3,
         select: %{
           item: u,
           exact_match: ilike(u.username, ^"%#{query_only_alphaneumeric}%"),
           cd_rank:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", u.searchable, ^query)
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              u.searchable,
+              ^query_websearch
+            )
         }
       )
 
     projects_query =
       from(
         p in Project,
         where:
-          fragment("? @@ websearch_to_tsquery('simple', ?)", p.searchable, ^query) or
+          fragment("? @@ websearch_to_tsquery('simple', ?)", p.searchable, ^query_websearch) or
             ilike(p.name, ^"%#{query_only_alphaneumeric}%"),
         join: pm in assoc(p, :memberships),
         on: pm.user_id == ^user.id,
@@ -139,23 +166,31 @@ defmodule Platform.GlobalSearch do
               ilike(p.name, ^"%#{query_only_alphaneumeric}%")
             ),
           desc:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", p.searchable, ^query),
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              p.searchable,
+              ^query_websearch
+            ),
           desc: p.inserted_at
         ],
         limit: 3,
         select: %{
           item: p,
           exact_match: ilike(p.name, ^"%#{query_only_alphaneumeric}%"),
           cd_rank:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", p.searchable, ^query)
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              p.searchable,
+              ^query_websearch
+            )
         }
       )
 
     updates_query =
       from(
         u in Update,
         where:
-          fragment("? @@ websearch_to_tsquery('simple', ?)", u.searchable, ^query) or
+          fragment("? @@ websearch_to_tsquery('simple', ?)", u.searchable, ^query_websearch) or
             ilike(u.explanation, ^"%#{query_only_alphaneumeric}%") or
             ilike(u.explanation, ^"%#{query_lower_raw}%"),
         join: m in assoc(u, :media),
@@ -171,7 +206,11 @@ defmodule Platform.GlobalSearch do
                 ilike(u.explanation, ^"%#{query_lower_raw}%")
             ),
           desc:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", u.searchable, ^query),
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              u.searchable,
+              ^query_websearch
+            ),
           desc: u.inserted_at
         ],
         limit: 3,
@@ -181,7 +220,11 @@ defmodule Platform.GlobalSearch do
             ilike(u.explanation, ^"%#{query_only_alphaneumeric}%") or
               ilike(u.explanation, ^"%#{query_lower_raw}%"),
           cd_rank:
-            fragment("ts_rank_cd(?, websearch_to_tsquery('simple', ?))", u.searchable, ^query)
+            fragment(
+              "ts_rank_cd(?, websearch_to_tsquery('simple', ?))",
+              u.searchable,
+              ^query_websearch
+            )
         }
       )
       |> Platform.Updates.preload_fields()
@@ -198,7 +241,7 @@ defmodule Platform.GlobalSearch do
           |> Enum.filter(fn item -> Permissions.can_view_media?(user, item.item) end)
         end),
         Task.async(fn ->
-          if String.length(query) < 3 do
+          if String.length(query_websearch) < 3 do
             []
           else
             Repo.all(users_query)