Merge pull request #2179 from obsidian-tasks-group/support-variables

feat: Support variables in custom filters and groups

docs/Introduction.md

@@ -6,6 +6,7 @@ publish: true
 
 ## What's New?
 
+- X.Y.Z: 🔥 Support [[Expressions#More complex expressions|variables, if statements, and functions]] in custom filters and groups
 - 4.3.0: 🔥 Bug fixes, usability improvements and `explain` support for [[Regular Expressions|regular expression]] searches
 - 4.2.0: 🔥 Add [[Custom Filters|custom filtering]]
 - 4.1.0: 🔥 Add [[Layout|hide and show tags]]

docs/Queries/Filters.md

@@ -293,7 +293,8 @@ Since Tasks 4.2.0, **[[Custom Filters|custom filtering]] by status symbol** is n
   - Find tasks with a checkbox `[-]`, which is conventionally used to mean "cancelled".
 - ```filter by function task.status.symbol !== ' '```
   - Find tasks with anything but the space character as their status symbol, that is, without the checkbox `[ ]`.
-- ```filter by function task.status.symbol === 'P' || task.status.symbol === 'C' || task.status.symbol === 'Q' || task.status.symbol === 'A'```
+- ```filter by function const symbol = task.status.symbol; return symbol === 'P' || symbol === 'C' || symbol === 'Q' || symbol === 'A'```
+  - Note that because we use a variable to avoid repetition, we need to add `return`
   - Find tasks with status symbol `P`, `C`, `Q` or `A`.
   - This can get quite verbose, the more symbols you want to search for.
 - ```filter by function 'PCQA'.includes(task.status.symbol)```
@@ -1021,11 +1022,12 @@ Since Tasks 4.2.0, **[[Custom Filters|custom filtering]] by heading** is now pos
 
 <!-- placeholder to force blank line before included text --> <!-- include: CustomFilteringExamples.test.file_properties_task.heading_docs.approved.md -->
 
-- ```filter by function task.due.moment?.isSame('2023-06-11', 'day') || ( !task.due.moment && task.heading?.includes('2023-06-11')) || false```
+- ```filter by function const taskDate = task.due.moment; const wanted = '2023-06-11'; return taskDate?.isSame(wanted, 'day') || ( !taskDate && task.heading?.includes(wanted)) || false```
   - Find takes that:
     - **either** due on the date `2023-06-11`,
     - **or** do not have a due date, and their preceding heading contains the same date as a string: `2023-06-11`.
-- ```filter by function task.due.moment?.isSame(moment(), 'day') || ( !task.due.moment && task.heading?.includes(moment().format('YYYY-MM-DD')) ) || false```
+  - Note that because we use variables to avoid repetition of values, we need to add `return`.
+- ```filter by function const taskDate = task.due.moment; const now = moment(); return taskDate?.isSame(now, 'day') || ( !taskDate && task.heading?.includes(now.format('YYYY-MM-DD')) ) || false```
   - Find takes that:
     - **either** due on today's date,
     - **or** do not have a due date, and their preceding heading contains today's date as a string, formatted as `YYYY-MM-DD`.

docs/Queries/Grouping.md

@@ -189,20 +189,23 @@ DON'T PANIC! For users who are comfortable with JavaScript, these more complicat
 - ```group by function task.due.format("[%%]d[%%]dddd")```
   - Group by day of the week (Sunday, Monday, Tuesday, etc).
   - The day names are sorted in date order, starting with Sunday.
-- ```group by function task.due.moment ? ( task.due.moment.day() === 0 ? task.due.format("[%%][8][%%]dddd") : task.due.format("[%%]d[%%]dddd") ) : "Undated"```
+- ```group by function const date = task.due; return date.moment ? ( date.moment.day() === 0 ? date.format("[%%][8][%%]dddd") : date.format("[%%]d[%%]dddd") ) : "Undated"```
   - Group by day of the week (Monday, Tuesday, etc).
   - The day names are sorted in date order, starting with Monday.
   - Tasks without due dates are displayed at the end, under a heading "Undated".
   - This is best understood by pasting it in to a Tasks block in Obsidian and then deleting parts of the expression.
   - The key technique is to say that if the day is Sunday (`0`), then force it to be displayed as date number `8`, so it comes after the other days of the week.
-- ```group by function (!task.due.moment) ? '%%4%% Undated' : result = task.due.moment.isBefore(moment(), 'day') ? '%%1%% Overdue' : result = task.due.moment.isSame(moment(), 'day') ? '%%2%% Today' : '%%3%% Future'```
+  - Note that because we use variables to avoid repetition of values, we need to add `return`
+- ```group by function const date = task.due.moment; return (!date) ? '%%4%% Undated' : date.isBefore(moment(), 'day') ? '%%1%% Overdue' : date.isSame(moment(), 'day') ? '%%2%% Today' : '%%3%% Future'```
   - Group task due dates in to 4 broad categories: `Overdue`, `Today`, `Future` and `Undated`, displayed in that order.
   - Try this on a line before `group by due` if there are a lot of due date headings, and you would like them to be broken down in to some kind of structure.
   - A limitation of Tasks expressions is that they each need to fit on a single line, so this uses nested ternary operators, making it powerful but very hard to read.
   - In fact, for ease of development and testing, it was written in a full-fledged development environment as a series of if/else blocks, and then automatically refactored in these nested ternary operators.
-- ```group by function (!task.due.moment) ? '%%4%% ==Undated==' : result = task.due.moment.isBefore(moment(), 'day') ? '%%1%% ==Overdue==' : result = task.due.moment.isSame(moment(), 'day') ? '%%2%% ==Today==' : '%%3%% ==Future=='```
+- ```group by function const date = task.due.moment; return (!date) ? '%%4%% ==Undated==' : date.isBefore(moment(), 'day') ? '%%1%% ==Overdue==' : date.isSame(moment(), 'day') ? '%%2%% ==Today==' : '%%3%% ==Future=='```
   - As above, but the headings `Overdue`, `Today`, `Future` and `Undated` are highlighted.
   - See the sample screenshot below.
+- ```group by function const date = task.due.moment; const now = moment(); const label = (order, name) => `%%${order}%% ==${name}==`; if (!date) return label(4, 'Undated'); if (date.isBefore(now, 'day')) return label(1, 'Overdue'); if (date.isSame(now, 'day')) return label(2, 'Today'); return label(3, 'Future');```
+  - As above, but using a local function, and `if` statements.
 
 <!-- placeholder to force blank line after included text --> <!-- endInclude -->
 

docs/Scripting/Expressions.md

@@ -20,6 +20,8 @@ publish: true
 
 - Language is JavaScript.
 - The expression is a string instruction.
+- It must fit on one line.
+  - As of Tasks X.Y.Z, variables, functions, `if` blocks and similar can be used. See [[#More complex expressions]].
 - Depending on the context, one or two tasks are passed in to the expression, and a calculation is performed.
   - As of Tasks 4.0.0, in fact only a single task is passed in, to implement [[Custom Grouping]].
   - As of Tasks 4.2.0, a single task is passed in, to implement [[Custom Filters]].
@@ -37,7 +39,9 @@ Each line below is of the form:
 expression => result
 ~~~
 
-Sample expressions:
+### Simple expressions
+
+Some example expressions:
 
 <!-- placeholder to force blank line before included text --> <!-- include: Expression.test.Expression_result.approved.md -->
 
@@ -70,3 +74,24 @@ Note:
 - Single quotes (`'`) and double quotes (`"`) are generally equivalent and you can use whichever you prefer.
 - The `||` means 'or'. If the expression to the left of the `||` fails, the expression on the right is used instead.
 - You can experiment with these values by adding them to a `group by function` line in a Tasks query block.
+- If you don't write a `return` statement, Tasks adds one for you.
+
+### More complex expressions
+
+As of Tasks X.Y.Z, it is also possible to use more complex constructs in expressions:
+
+- `return` statements
+- named variables
+- `if` statements
+- functions
+
+<!-- placeholder to force blank line before included text --> <!-- include: Expression.test.Expression_returns_and_functions.approved.md -->
+
+~~~text
+return 42 => 42
+const x = 1 + 1; return x * x => 4
+if (1 === 1) { return "yes"; } else { return "no" } => 'yes'
+function f(value) { if (value === 1 ) { return "yes"; } else { return "no"; } } return f(1) => 'yes'
+~~~
+
+<!-- placeholder to force blank line after included text --> <!-- endInclude -->

src/Scripting/Expression.ts

@@ -30,7 +30,8 @@ export function constructArguments(task: Task | null) {
 export function parseExpression(paramsArgs: [string, any][], arg: string): FunctionOrError {
     const params = paramsArgs.map(([p]) => p);
     try {
-        const expression: '' | null | Function = arg && new Function(...params, `return ${arg}`);
+        const input = arg.includes('return') ? arg : `return ${arg}`;
+        const expression: '' | null | Function = arg && new Function(...params, input);
         if (expression instanceof Function) {
             return FunctionOrError.fromObject(arg, expression);
         }

tests/CustomMatchers/CustomMatchersForExpressions.ts

@@ -0,0 +1,31 @@
+import { diff } from 'jest-diff';
+import { evaluateExpression, parseExpression } from '../../src/Scripting/Expression';
+
+declare global {
+    namespace jest {
+        interface Matchers<R> {
+            toEvaluateAs(expected: any): CustomMatcherResult;
+        }
+    }
+}
+
+// Based on https://stackoverflow.com/a/60229956/104370
+export function toEvaluateAs(instruction: string, expected: any): jest.CustomMatcherResult {
+    const functionOrError = parseExpression([], instruction);
+    expect(functionOrError.queryComponent).not.toBeUndefined();
+    const received = evaluateExpression(functionOrError.queryComponent!, []);
+
+    const pass: boolean = received === expected;
+    const expectedAsText = expected.toString();
+    const receivedAsText = received ? received.toString() : 'null';
+
+    const message: () => string = () =>
+        pass
+            ? `Expression result should not be ${expectedAsText}`
+            : `Expression result is not the same as expected: ${diff(expectedAsText, receivedAsText)}`;
+
+    return {
+        message,
+        pass,
+    };
+}

tests/CustomMatchers/jest.custom_matchers.setup.ts

@@ -6,6 +6,14 @@ expect.extend({
     toEqualMoment,
 });
 
+// ---------------------------------------------------------------------
+// CustomMatchersForExpressions
+// ---------------------------------------------------------------------
+import { toEvaluateAs } from './CustomMatchersForExpressions';
+expect.extend({
+    toEvaluateAs,
+});
+
 // ---------------------------------------------------------------------
 // CustomMatchersForFilters
 // ---------------------------------------------------------------------

tests/Scripting/Expression.test.Expression_returns_and_functions.approved.md

@@ -0,0 +1,11 @@
+<!-- placeholder to force blank line before included text -->
+
+~~~text
+return 42 => 42
+const x = 1 + 1; return x * x => 4
+if (1 === 1) { return "yes"; } else { return "no" } => 'yes'
+function f(value) { if (value === 1 ) { return "yes"; } else { return "no"; } } return f(1) => 'yes'
+~~~
+
+
+<!-- placeholder to force blank line after included text -->

tests/Scripting/Expression.test.ts

@@ -17,16 +17,48 @@ import { formatToRepresentType } from './ScriptingTestHelpers';
 window.moment = moment;
 
 describe('Expression', () => {
-    const task = TaskBuilder.createFullyPopulatedTask();
+    describe('support simple calculations', () => {
+        it('should calculate simple expression', () => {
+            expect('1 + 1').toEvaluateAs(2);
+        });
 
-    describe('detect errors at parse stage', () => {
-        it('should report meaningful error message for duplicate return statement', () => {
-            // evaluateExpression() currently adds a return statement, to save user typing it.
-            expect(parseExpression([], 'return 42').error).toEqual(
-                'Error: Failed parsing expression "return 42".\nThe error message was:\n    "SyntaxError: Unexpected token \'return\'"',
-            );
+        it('should support return statements', () => {
+            expect('return 42').toEvaluateAs(42);
         });
 
+        it('should allow use of a variable in expression', () => {
+            expect('const x = 1 + 1; return x;').toEvaluateAs(2);
+        });
+
+        it('should support if blocks', () => {
+            expect('if (1 === 1) { return "yes"; } else { return "no"; }').toEvaluateAs('yes');
+            expect('if (1 !== 1) { return "yes"; } else { return "no"; }').toEvaluateAs('no');
+        });
+
+        it('should support functions - multi-line', () => {
+            // Tasks only supports single-line expressions.
+            // This multi-line one is used for readability
+            const line = `
+                function f(value) {
+                    if (value === 1 ) {
+                        return "yes";
+                    } else {
+                        return "no";
+                    }
+                }
+                return f(1)`;
+            expect(line).toEvaluateAs('yes');
+        });
+
+        it('should support functions - single-line', () => {
+            const line = 'function f(value) { if (value === 1 ) { return "yes"; } else { return "no"; } } return f(1)';
+            expect(line).toEvaluateAs('yes');
+        });
+    });
+
+    const task = TaskBuilder.createFullyPopulatedTask();
+
+    describe('detect errors at parse stage', () => {
         it('should report meaningful error message for parentheses too few parentheses', () => {
             expect(parseExpression([], 'x(').error).toEqual(
                 'Error: Failed parsing expression "x(".\nThe error message was:\n    "SyntaxError: Unexpected token \'}\'"',
@@ -66,6 +98,16 @@ describe('Expression', () => {
         });
     });
 
+    function verifyExpressionsForDocs(expressions: string[]) {
+        let markdown = '~~~text\n';
+        for (const expression of expressions) {
+            const result = parseAndEvaluateExpression(task, expression);
+            markdown += `${expression} => ${formatToRepresentType(result)}\n`;
+        }
+        markdown += '~~~\n';
+        verifyMarkdownForDocs(markdown);
+    }
+
     it('result', () => {
         const expressions = [
             "'hello'",
@@ -89,13 +131,16 @@ describe('Expression', () => {
             // Should allow manual escaping of markdown
             String.raw`"I _am_ not _italic_".replaceAll("_", "\\_")`,
         ];
+        verifyExpressionsForDocs(expressions);
+    });
 
-        let markdown = '~~~text\n';
-        for (const expression of expressions) {
-            const result = parseAndEvaluateExpression(task, expression);
-            markdown += `${expression} => ${formatToRepresentType(result)}\n`;
-        }
-        markdown += '~~~\n';
-        verifyMarkdownForDocs(markdown);
+    it('returns and functions', () => {
+        const expressions = [
+            'return 42',
+            'const x = 1 + 1; return x * x',
+            'if (1 === 1) { return "yes"; } else { return "no" }',
+            'function f(value) { if (value === 1 ) { return "yes"; } else { return "no"; } } return f(1)',
+        ];
+        verifyExpressionsForDocs(expressions);
     });
 });

tests/Scripting/ScriptingReference/CustomFiltering/CustomFilteringExamples.test.file_properties_task.heading_docs.approved.md

@@ -1,10 +1,11 @@
 <!-- placeholder to force blank line before included text -->
 
-- ```filter by function task.due.moment?.isSame('2023-06-11', 'day') || ( !task.due.moment && task.heading?.includes('2023-06-11')) || false```
+- ```filter by function const taskDate = task.due.moment; const wanted = '2023-06-11'; return taskDate?.isSame(wanted, 'day') || ( !taskDate && task.heading?.includes(wanted)) || false```
     - Find takes that:
       - **either** due on the date `2023-06-11`,
       - **or** do not have a due date, and their preceding heading contains the same date as a string: `2023-06-11`.
-- ```filter by function task.due.moment?.isSame(moment(), 'day') || ( !task.due.moment && task.heading?.includes(moment().format('YYYY-MM-DD')) ) || false```
+    - Note that because we use variables to avoid repetition of values, we need to add `return`.
+- ```filter by function const taskDate = task.due.moment; const now = moment(); return taskDate?.isSame(now, 'day') || ( !taskDate && task.heading?.includes(now.format('YYYY-MM-DD')) ) || false```
     - Find takes that:
       - **either** due on today's date,
       - **or** do not have a due date, and their preceding heading contains today's date as a string, formatted as `YYYY-MM-DD`.

tests/Scripting/ScriptingReference/CustomFiltering/CustomFilteringExamples.test.file_properties_task.heading_results.approved.txt

@@ -2,17 +2,18 @@ Results of custom filters
 
 
 
-filter by function task.due.moment?.isSame('2023-06-11', 'day') || ( !task.due.moment && task.heading?.includes('2023-06-11')) || false
+filter by function const taskDate = task.due.moment; const wanted = '2023-06-11'; return taskDate?.isSame(wanted, 'day') || ( !taskDate && task.heading?.includes(wanted)) || false
 Find takes that:
   **either** due on the date `2023-06-11`,
   **or** do not have a due date, and their preceding heading contains the same date as a string: `2023-06-11`.
+Note that because we use variables to avoid repetition of values, we need to add `return`.
 =>
 - [ ] Due on 2023-06-11 📅 2023-06-11
 - [ ] No due date but I have 2023-06-11 in my preceding heading
 ====================================================================================
 
 
-filter by function task.due.moment?.isSame(moment(), 'day') || ( !task.due.moment && task.heading?.includes(moment().format('YYYY-MM-DD')) ) || false
+filter by function const taskDate = task.due.moment; const now = moment(); return taskDate?.isSame(now, 'day') || ( !taskDate && task.heading?.includes(now.format('YYYY-MM-DD')) ) || false
 Find takes that:
   **either** due on today's date,
   **or** do not have a due date, and their preceding heading contains today's date as a string, formatted as `YYYY-MM-DD`.

tests/Scripting/ScriptingReference/CustomFiltering/CustomFilteringExamples.test.statuses_task.status.symbol_docs.approved.md

@@ -4,7 +4,8 @@
     - Find tasks with a checkbox `[-]`, which is conventionally used to mean "cancelled".
 - ```filter by function task.status.symbol !== ' '```
     - Find tasks with anything but the space character as their status symbol, that is, without the checkbox `[ ]`.
-- ```filter by function task.status.symbol === 'P' || task.status.symbol === 'C' || task.status.symbol === 'Q' || task.status.symbol === 'A'```
+- ```filter by function const symbol = task.status.symbol; return symbol === 'P' || symbol === 'C' || symbol === 'Q' || symbol === 'A'```
+    - Note that because we use a variable to avoid repetition, we need to add `return`
     - Find tasks with status symbol `P`, `C`, `Q` or `A`.
     - This can get quite verbose, the more symbols you want to search for.
 - ```filter by function 'PCQA'.includes(task.status.symbol)```

tests/Scripting/ScriptingReference/CustomFiltering/CustomFilteringExamples.test.statuses_task.status.symbol_results.approved.txt

@@ -25,7 +25,8 @@ Find tasks with anything but the space character as their status symbol, that is
 ====================================================================================
 
 
-filter by function task.status.symbol === 'P' || task.status.symbol === 'C' || task.status.symbol === 'Q' || task.status.symbol === 'A'
+filter by function const symbol = task.status.symbol; return symbol === 'P' || symbol === 'C' || symbol === 'Q' || symbol === 'A'
+Note that because we use a variable to avoid repetition, we need to add `return`
 Find tasks with status symbol `P`, `C`, `Q` or `A`.
 This can get quite verbose, the more symbols you want to search for.
 =>