Hanging after hit save (:w) with emacs 28

[reyman]

Even running with -debug-init, emacs Freezing/Hanging problem when saving (:w) with transclude mode running : garbage collecting running infinitly.

Emacs
GNU Emacs 28.0.90 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0) of 2021-12-15

Org file transcluded (my-file.org) :

#+name: prepare-secrets
#+header: :var FILE=""
#+begin_src sh :noweb yes :exports result
if [ -f "$FILE" ]; then
    echo "$FILE already exist"
else
    publicKey="/home/xxx/xxx.asc"
    dd if=/dev/urandom bs=1024 count=4 | base64 --wrap=0 | gpg --batch --recipient-file $publicKey -o xxx.asc -a --encrypt --yes
fi
#+end_src

Into main.org, call to function in the transcluded file :

my-file.org
#+transclude: [[file:myfile.org]]

#+CALL: prepare-secrets(FILE="/home/xxx/xxx.asc")

Doom version

v21.12.0-alpha

A debug trace i have sometimes when i hit :w

 Debugger entered--Lisp error: (doom-hook-error org-transclusion-after-save-buffer (user-error "Not at a transclude keyword"))
  signal(doom-hook-error (org-transclusion-after-save-buffer (user-error "Not at a transclude keyword")))
  (condition-case e (funcall hook) ((debug error) (signal 'doom-hook-error (list hook e))))
  doom-run-hook(org-transclusion-after-save-buffer)
  run-hook-wrapped(doom-run-hook org-transclusion-after-save-buffer)
  (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car (cdr e))) (symbol-name (car (cdr e))) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e)))))
  (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car ...)) (symbol-name (car ...)) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp ...) (symbol-name ...) (car ...)) (car (cdr ...)))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- hooks)) (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if ... ... ...) (car ...))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  doom-run-hooks(after-save-hook)
  apply(doom-run-hooks after-save-hook)
  run-hooks(after-save-hook)
  basic-save-buffer(nil)
  save-buffer()
  evil-write(nil nil nil nil nil)
  funcall-interactively(evil-write nil nil nil nil nil)
  call-interactively(evil-write)
  evil-ex-call-command(nil #("w" 0 1 (ex-index 1)) nil)
  eval((evil-ex-call-command nil #("w" 0 1 (ex-index 1)) nil))
  evil-ex-execute(#("w" 0 1 (ex-index 1)))
  #<subr evil-ex>(nil)
  apply(#<subr evil-ex> nil)
  evil-ex(nil)
  funcall-interactively(evil-ex nil)
  command-execute(evil-ex)

[nobiot]

Thank you for reporting.
I copy-and-pasted both of your files and tried saving both after transclusion has been added.
I don't know if I'm doing this correctly, but I cannot reproduce the problem.

This screen shot was taken from GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.30, cairo version 1.16.0) of 2021-11-13 (native-compile). I will try with Windows and Emacs 27.2 later.

Is it possible for you to try without Doom (e.g. I have heard it has a sandbox system to remove Doom configuration) by any chance?

[reyman]

Thanks @nobiot i finally found a way to dehang / defreeze running pkill -USR2 emacs :

Debugger entered--entering a function:
* #f(compiled-function () #<bytecode 0xb4f4fa2509a17>)()
  org-element--parse-to(160)
  org-element-cache-map(ignore :granularity headline+inlinetask)
  org-element-cache-map(#f(compiled-function (el) #<bytecode 0x1873ada1b94e8481>) :next-re "^\\*+ " :fail-re "^\\*+ " :narrow t)
  org-scan-tags(org-encrypt-entry (lambda (todo tags-list level) (progn (setq org-cached-props nil) (or (and (member "crypt" tags-list))))) nil)
  org-encrypt-entries()
  funcall(org-encrypt-entries)
  (condition-case e (funcall hook) ((debug error) (signal 'doom-hook-error (list hook e))))
  doom-run-hook(org-encrypt-entries)
  run-hook-wrapped(doom-run-hook org-encrypt-entries)
  (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car (cdr e))) (symbol-name (car (cdr e))) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e)))))
  (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp (car ...)) (symbol-name (car ...)) (car (cdr e))) (car (cdr (cdr e))))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if (symbolp ...) (symbol-name ...) (car ...)) (car (cdr ...)))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- hooks)) (while --dolist-tail-- (let ((hook (car --dolist-tail--))) (condition-case e (run-hook-wrapped hook #'doom-run-hook) ((debug doom-hook-error) (if debug-on-error nil (lwarn hook :error "Error running hook %S because: %s" (if ... ... ...) (car ...))) (signal 'doom-hook-error (cons hook (cdr e))))) (setq --dolist-tail-- (cdr --dolist-tail--)))))
  doom-run-hooks(before-save-hook)
  apply(doom-run-hooks before-save-hook)
  run-hooks(before-save-hook)
  basic-save-buffer(nil)
  save-buffer()
  evil-write(nil nil nil nil nil)
  #<subr funcall-interactively>(evil-write nil nil nil nil nil)
  apply(#<subr funcall-interactively> (evil-write nil nil nil nil nil))
  funcall-interactively(evil-write nil nil nil nil nil)
  #<subr call-interactively>(evil-write)
  apply(#<subr call-interactively> evil-write)
  explain-pause--wrap-call-interactively(#<subr call-interactively> evil-write)
  apply(explain-pause--wrap-call-interactively #<subr call-interactively> evil-write)
  call-interactively(evil-write)
  evil-ex-call-command(nil #("w" 0 1 (ex-index 1)) nil)
  eval((evil-ex-call-command nil #("w" 0 1 (ex-index 1)) nil))
  evil-ex-execute(#("w" 0 1 (ex-index 1)))
  evil-ex(nil)
  #<subr funcall-interactively>(evil-ex nil)
  apply(#<subr funcall-interactively> (evil-ex nil))
  funcall-interactively(evil-ex nil)
  #<subr call-interactively>(evil-ex nil nil)
  apply(#<subr call-interactively> (evil-ex nil nil))
  explain-pause--wrap-call-interactively(#<subr call-interactively> evil-ex nil nil)
  apply(explain-pause--wrap-call-interactively #<subr call-interactively> (evil-ex nil nil))
  call-interactively(evil-ex nil nil)
  command-execute(evil-ex)

I try without doom if i found way to do that.

[nobiot]

It could be also related to #105 -- the issue will start to appear with Org v9.6, which Doom pulls in. I'm still using 9.5 and 9.4 (9.6 has not been released yet).

[nobiot]

Windows seems fine to me.

• GNU Emacs 27.2 (build 1, x86_64-w64-mingw32) of 2021-03-26
• Org mode version 9.4.4 (release_9.4.4 @ c:/Program Files/Emacs/share/emacs/27.2/lisp/org/)

So I think it's either Org 9.6 or Doom (or some interaction between them with Org-transclusion).

[reyman]

I try with doom sandbox mode with vanilla, but that create other errors because langage in org-babel are not loaded automatically outside doom, i will retry tonight after reading some doc.

I try to rebuild doom + emacs 28 + pin to previous org mode to see (https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tag/?h=release_9.5)

Edit : I have errors when i !pin org to another version ... so it seems this is not a good pratice : doomemacs/doomemacs#5721 say @hlissner

I retry by rebuilding

[reyman]

@nobiot I confirm even with doom sandbox and emacs vanilla, i cannot use org-transclusion :-/

Transcluded code is correctly added to buffer in READ-MODE, but i can't execute M-X org-babel-execute-buffer, because emacs says "Text is read only" for transcluded text.

That don't freeze, but that also don't work "snif ..."

[nobiot]

I think that's the limitation noted in readme -- it's the issue with noweb.

Org-transclusion does not support expansion of noweb references when a transcluded source block code has them
Refer to issue #86[fn:6:https://github.com//issues/86]. You will get “Text read-only” error when export tries to expand the noweb references into the source code. †noweb reference[fn:7:https://orgmode.org/manual/Noweb-Reference-Syntax.html]

I don't use it and don't really know what it does. Is it possible for you to not use it?

It seems to try to replace the transcluded region, which it cannot because the region is read-only. You might be able to temporarily disable read-only as discussed in issue #86 for export.

[reyman]

I don't use it and don't really know what it does. Is it possible for you to not use it?

Noweb is a really powerfull functionality present originaly in Literate Programing of Donald Knuth. There is two possibility, the <<block-to-replace>> are expanded by replacing the <<>> by block designed using #+NAME:block-to-replace during tangle or evaluation. Another possibility was to replace not by the content but by the result of the #+NAME:block-to-replace evaluation, by calling <<block-to-replace()>>

You have very simple example here : https://orgmode.org/manual/Noweb-Reference-Syntax.html

And for real example of usage :
https://filesender.renater.fr/?s=download&token=f6f28a9b-7179-4669-b66a-fa8caee24e91

I will try by removing noweb

[nobiot]

@reyman

I think I have made noweb to work with the latest commit.

See below.

If you have time, I'd appreciate if you tested this with the latest commit.
I don't use R, so I cannot really use your real example -- thank you for sending it over, though :)

test.org is this (copied from https://orgmode.org/manual/Noweb-Reference-Syntax.html)

#+NAME: initialization
#+BEGIN_SRC emacs-lisp
  (setq sentence "Never a foot too far, even.")
#+END_SRC

#+BEGIN_SRC emacs-lisp :noweb yes
  <<initialization>>
  (reverse sentence)
#+END_SRC

Org-transclusion is part of ELPA. There is devel for unreleased packages: https://elpa.gnu.org/devel/org-transclusion.html
It will take some time to sync the latest with GitHub with even the devel.

To use ELPA-devel, you would need to add this to variable package-archives

("gnu-devel" . "https://elpa.gnu.org/devel/")

For Doom, it might be easier to get the source from GitHub.

[reyman]

@reyman

I think I have made noweb to work with the latest commit.

See below.
...

Sorry for the delay, i will try before or after the 1/1/22

Happy new year and thanks again for this package ;)

[nobiot]

@reyman
Likewise :)

The change is now pushed to the latest (stable) ELPA release http://elpa.gnu.org/packages/org-transclusion.html so I believe you can simply use package! and doom sync to upgrade Org-transclusion. Let me know how you go (but please take all the time you need and enjoy the holiday season :))

[ugurbolat]

I've rolled back to Org 9.5 but I still experience the issue when saving buffer.

it is duplicating the links...

Update: my issue turned out to be caused by real-auto-save, which doesn't have much functionality to be causing a problem...

Update 2: it seems like it is the combination of org 9.6 and real-auto-save...

[nobiot]

@ugurbolat

This is certainly a critical issue and I would love to reproduce and fix it.

I'd appreciate it if you can:

• tell me more detail about how to get to the error
• create a new issue as the original seems to be a different error
• ensure you have the latest version of Org-transclusion (what's your version?)

At the moment, I cannot reproduce it. I just tried with the following:

• Ubuntu Linux
• Emacs version 28.1.50
• org-9.6pre0.20221024.43732
• Startt emacs with "-q" option in terminal and evaluate the following in the scratch buffer
• In a org buffer, I activate real-save-buffer-mode, activate a transclusion, make sure the buffer is changed, move to another window, and wait for 10 seconds
• I confirm the org-buffer gets saved
• No problem observed

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
(add-to-list 'package-archives '("gnu-devel" . "https://elpa.gnu.org/devel/") t)

(package-initialize)
(package-refresh-contents)

(package-install 'org-transclusion)
(package-install 'real-auto-save)

(define-key global-map (kbd "<f12>") #'org-transclusion-add)
```

[nobiot]

In order to contain the issue of infinite loop, I have pushed commit 43c478c. I'd appreciate it if anyone has bumped into the error message: "org-transclusion: Aborting. You may be in an infinite loop".

fix: heuristics to identify & break infinite loop on save

Reported in GitHub issues #109 #177.

I cannot reproduce the issue myself so far. I am putting in place (1)
small preventive measure and (2) heuristics to defect and break the
infinite loop on save-buffer.

(1) Org-transclusion (OT) tries not to save the transcluded buffer
content and instead save only the #+transclude keyword line to the file.
To achieve this, OT uses 'before-' and 'after-save-hook' to remove-all
the transclusions and then add-all them. This operation relies on the
returned value of the point from 'org-transclusion-remove' function. In
this commit, the point (integer) is changed to marker. This way, any
arbitrary buffer change between these remove-all and add-all processes
can have less impact on the moving points of reference -- makers
automatically move to adopt to the new buffer state. I suspect something
like 'whitespace-cleanup` put in 'before-save-buffer' might dislocate
the positions in some situations. This preventive measure hopefully
preempt the issues.

(2) The heuristics is simple but should work if there is an unexpected
number loop happens. Since it is simply compare the length of a list,
and the 'dolist' loops for the same list, logically this should be
redundant; however, since the infinite loop itself to me is anomaly,
this heuristics might catch the issue and break the loop.

As you can see, both attempts are not based on causal analysis but rather
"stabbing in the dark" heuristics.