Executing blocks with malformed results section erases org file contents

[pati-ni]

In some scenarios (does not happen often) the executing a src block erases the appended commend of the org-mode file. The result is that everything at least until the next src block gets erased.

My guess is that in the #+RESULTS: section the :RESULTS: are not properly wrapped with the :END: block. Thus, re-executing the src block erases everything up until the next :END: block. If lucky and executed the next block it will stop in :END: of that block. If not, most likely it will erase everything. Usually this can be reverted, if I realize described behavior with an undo.

But this can lead to some nasty situations. Is there a way to introduce a fail-safe for this?

[ed9w2in6]

I face a similar symptom regularly but only on commands that fails and triggers error / warnings.
In which org-mode will complain, here are an excerpt of my *Warnings* buffer

 ■  Warning (org-element-cache): org-element--cache: Unregistered buffer modifications detected (312536 != 312301). Resetting.
If this warning appears regularly, please report the warning text to Org mode mailing list (M-x org-submit-bug-report).
The buffer is: scratch.org
 Current command: nil
 Backtrace:
nil

So a good starting point on any investigation would be:

• how emacs-jupyter put error message ([goto-error]?) into org buffer
  • via jupyter-handle-error
• how org-element--cache-sync expects buffer are edited.

Sorry that I do not have time right now to look further into this but I shall investigate more on it later since it affects me a lot.
I used to just ignore it but lately I found that huge sections of my org file got deleted away without me noticing.
Undo also won't work but luckily I had made a backup few weeks ago.

[nnicandro]

Could you give an example where Emacs-Jupyter causes the :END: line to be missing from the results of a source block? There shouldn't be any such cases happening, if there are let me know.

It would be great if you could provide a minimum working example Org file with example source blocks that can reproduce your problem. I am aware of the scenario that you mention, but evaluating source blocks with Emacs-Jupyter should always create valid Org documents.

[akirakyle]

I have also been seeing these issues but haven't had a chance to debug further and create a minimum working example. I suspect the org-element--cache issues may be related to to the way jupyter handles ansi colors as I still see inconsistent behavior for error outputs that have escape characters to color them.

[ed9w2in6]

@nnicandro
For me, I am not sure if :END: line is missing all the time when erasure of contents happen.
I am sure that:

• if the code fails and triggers error / warnings, content MIGHT be erased.
• if it triggers error, THEN try doing undo will DEFINITELY erase content, due to corrupted cache as shown in my previous comment (Executing blocks with malformed results section erases org file contents #486 (comment))

I can reproduce this quite reliably in my current setup so I'll try look into it maybe at the weekends for a minimal setup.

[NightMachinery]

I have also seen this happen, e.g.,

* before
#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both
8
#+end_src

#+RESULTS:
#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both
9
#+end_src

* after
#+begin_src jupyter-python :kernel py_base :session emacs_py_1 :async yes :exports both
8
#+end_src

#+RESULTS:
: 8

Here the problem is that I have deleted the blank line after #+RESULTS, but I think emacs-jupyter should be robust to such user misbehaviors. Can't it just check for the next #+begin_src and always stop there? After all, all results are prepended with : , so there should never be a line in the results that starts with #+begin_src. We can even just stop at the first #+.

[ed9w2in6]

@nnicandro Hmm, beware that mine might be a different issue. My issue have strong connection with undoing and `org-element--cache`. @NightMachinery Have you ever seen the warning messages from `org-element--cache`?

[NightMachinery]

@ed9w2in6 You’re right, the issue I raised is distinct from yours, but perhaps the solution I proposed would work for your case, as well?

[nnicandro]

@NightMachinery In Org, src blocks can also be the results of execution of some other source block, e.g.

#+begin_src shell :wrap "src jupyter-python"
echo 9
#+end_src

Running the above yields

#+begin_src shell :wrap "src jupyter-python"
echo 9
#+end_src

#+RESULTS:
#+begin_src jupyter-python
9
#+end_src

Org is the one that is removing the source block with the #+RESULTS keyword attached to it, not Emacs-Jupyter, when a new result is inserted see org-babel-insert-result. There is really no way to work around this unless we mess with Org's internals.

[NightMachinery]

@nnicandro I think this problem is severe enough that it warrants forking org-babel-insert-result. I am pretty sure I have lost code to this behavior.

This org feature of inserting a source block as a result of another thing is not that useful (I would personally use eval instead in such a situation, which will work even if I migrate to a script).

We can of course check :wrap and do delete begin_src if such a :wrap is present. This new behavior can even be sent upstream, no?

[mankoff]

Just to confirm that I can reliably reproduce this. Evaluate buffer with error. An example is:

#+BEGIN_SRC jupyter-python :exports both
import numpy as np
import xarray as xr
import pandas as pd

# foo = xr.Dataset({'foo': xr.DataArray(data=[100,200], dims=['dim'], coords={'dim':['a','b']})})
# bar = xr.Dataset({'bar': xr.DataArray(data=[200,300], dims=['dim'], coords={'dim':['b','c']})})
# baz = xr.Dataset({'baz': xr.DataArray(data=[200,100], dims=['dim'], coords={'dim':['b','a']})})
# print((foo['foo']-bar['bar']).values)
# print((foo['foo'] - baz['baz']).values)

times = pd.date_range(start='2000-01-01',freq='1D',periods=3)
dims = np.array(['aa','ab']).astype(np.object)

foo = xr.Dataset({'foo': xr.DataArray(data=[[1,2],[3,4],[5,6]], dims=['time','dim'], coords={'time':times, 'dim':dims})})
bar = xr.Dataset({'bar': xr.DataArray(data=[[2,1],[4,3],[6,5]], dims=['time','dim'], coords={'time':times, 'dim':dims[::-1]})})
# print((foo['foo']-bar['bar']).values)

ds = xr.Dataset()
ds['time'] = (('time'), times)
ds['dim'] = (('dim'), dims)
ds['baz'] = (('time','dim'), foo['foo']-bar['bar'])
print(ds)
#+END_SRC

Which errors with:

#+RESULTS:
:RESULTS:
: /tmp/ipykernel_521850/3839449454.py:12: FutureWarning: In the future `np.object` will be defined as the corresponding NumPy scalar.
:   dims = np.array(['aa','ab']).astype(np.object)
# [goto error]
#+begin_example
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[13], line 12
      5 # foo = xr.Dataset({'foo': xr.DataArray(data=[100,200], dims=['dim'], coords={'dim':['a','b']})})
      6 # bar = xr.Dataset({'bar': xr.DataArray(data=[200,300], dims=['dim'], coords={'dim':['b','c']})})
      7 # baz = xr.Dataset({'baz': xr.DataArray(data=[200,100], dims=['dim'], coords={'dim':['b','a']})})
      8 # print((foo['foo']-bar['bar']).values)
      9 # print((foo['foo'] - baz['baz']).values)
     11 times = pd.date_range(start='2000-01-01',freq='1D',periods=3)
---> 12 dims = np.array(['aa','ab']).astype(np.object)
     14 foo = xr.Dataset({'foo': xr.DataArray(data=[[1,2],[3,4],[5,6]], dims=['time','dim'], coords={'time':times, 'dim':dims})})
     15 bar = xr.Dataset({'bar': xr.DataArray(data=[[2,1],[4,3],[6,5]], dims=['time','dim'], coords={'time':times, 'dim':dims[::-1]})})

File ~/local/mambaforge/envs/ds/lib/python3.10/site-packages/numpy/__init__.py:305, in __getattr__(attr)
    300     warnings.warn(
    301         f"In the future `np.{attr}` will be defined as the "
    302         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
    304 if attr in __former_attrs__:
--> 305     raise AttributeError(__former_attrs__[attr])
    307 # Importing Tester requires importing all of UnitTest which is not a
    308 # cheap import Since it is mainly used in test suits, we lazy import it
    309 # here to save on the order of 10 ms of import time for most users
    310 #
    311 # The previous way Tester was imported also had a side effect of adding
    312 # the full `numpy.testing` namespace
    313 if attr == 'testing':

AttributeError: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`. To avoid this error in existing code, use `object` by itself. Doing this will not modify any behavior and is safe. 
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
#+end_example
:END:

Note that there is both an #+end_example and :END:.

However, if I then type C-/ which is (undo), it deletes the next ~100 lines of the buffer. Sections, babel blocks with BEGIN_SRC, etc. (a comment above said to the next source block but I am seeing this bug consume multiple source blocks).

I can reliably recreate this bug with my (large) config. I cannot create an MWE.

[ed9w2in6]

@mankoff The symptoms that you just described are exactly the same as mine.
Do you see the org-element--cache warning messages?

@nnicandro Do you prefer me and @mankoff to open another issue? At least the symptoms are different, if it turns out to be the same cause we can just close one of them.

[mankoff]

@ed9w2in6 I'm running latest org from git and updated yesterday. I cannot recreate this bug today. The git log has a lot of entries on org-element and the cache. Can you check if you still have this bug on the latest commit? I'm at

• 098f08159 - (HEAD -> main, origin/main, origin/HEAD) org-open-at-point: Preserve point unless opening link moves the point (2023-10-23)

[mankoff]

@ed9w2in6 I'm running latest org from git and updated yesterday. I cannot recreate this bug today. The git log has a lot of entries on org-element and the cache. Can you check if you still have this bug on the latest commit?

Nevermind, Still here :(.

[mankoff]

I posted about this issue on the Org list in case it is related to org-cache.

Their suggestion:

This is most likely a problem with emacs-jupyter. It does something that bypasses `after-change-functions', which is not allowed in Org mode.

[mankoff]

@mankoff The symptoms that you just described are exactly the same as mine.
Do you see the org-element--cache warning messages?

Yes, I get the same cache warning. I see

Toggle...

Warning (org-element-cache): org-element--cache: Unregistered buffer modifications detected (57640 != 55418). Resetting.
If this warning appears regularly, please report the warning text to Org mode mailing list (M-x org-submit-bug-report).
The buffer is: misc.org
 Current command: nil
 Backtrace:
"  backtrace-to-string(nil)
  org-element--cache-sync(# 37660)
  org-element-at-point()
  (progn (org-element-at-point))
  (unwind-protect (progn (org-element-at-point)) (set-match-data save-match-data-internal 'evaporate))
  (let ((save-match-data-internal (match-data))) (unwind-protect (progn (org-element-at-point)) (set-match-data save-match-data-internal 'evaporate)))
  (let ((element (let ((save-match-data-internal (match-data))) (unwind-protect (progn (org-element-at-point)) (set-match-data save-match-data-internal 'evaporate))))) (and (eq (org-element-type element) 'src-block) (>= (line-beginning-position) (let* ((parray (and t (let* ... ...)))) (if parray (let* ((val ...)) (if (eq val ...) 'nil (let ... val))) (let* ((val ...)) (cond (... ...) (... ...) (t ...)))))) (<= (line-end-position) (save-excursion (save-restriction (widen) (goto-char (let* (...) (if parray ... ...))) (skip-chars-backward \" \\11\\n\") (line-end-position)))) (org-element--property :language element nil nil)))
  org-eldoc-get-src-lang()
  (let ((lang (org-eldoc-get-src-lang))) (cond ((string= lang \"org\") nil) ((or (string= lang \"emacs-lisp\") (string= lang \"elisp\")) (cond ((and (boundp 'eldoc-documentation-functions) (fboundp 'elisp-eldoc-var-docstring) (fboundp 'elisp-eldoc-funcall)) (let ((eldoc-documentation-functions ...)) (eldoc-print-current-symbol-info))) ((fboundp 'elisp-eldoc-documentation-function) (elisp-eldoc-documentation-function)) (t (let (eldoc-documentation-function) (eldoc-print-current-symbol-info))))) ((or (string= lang \"c\") (string= lang \"C\")) (if (require 'c-eldoc nil t) (progn (c-eldoc-print-current-symbol-info)))) ((string= lang \"css\") (if (require 'css-eldoc nil t) (progn (css-eldoc-function)))) ((string= lang \"php\") (if (require 'php-eldoc nil t) (progn (php-eldoc-function)))) ((or (string= lang \"go\") (string= lang \"golang\")) (if (require 'go-eldoc nil t) (progn (go-eldoc--documentation-function)))) (t (let ((doc-fun (org-eldoc-get-mode-local-documentation-function lang)) (callback (car args))) (if (functionp doc-fun) (progn (if (functionp callback) (funcall doc-fun callback) (funcall doc-fun))))))))
  (or (org-eldoc-get-breadcrumb) (org-eldoc-get-src-header) (let ((lang (org-eldoc-get-src-lang))) (cond ((string= lang \"org\") nil) ((or (string= lang \"emacs-lisp\") (string= lang \"elisp\")) (cond ((and (boundp ...) (fboundp ...) (fboundp ...)) (let (...) (eldoc-print-current-symbol-info))) ((fboundp 'elisp-eldoc-documentation-function) (elisp-eldoc-documentation-function)) (t (let (eldoc-documentation-function) (eldoc-print-current-symbol-info))))) ((or (string= lang \"c\") (string= lang \"C\")) (if (require 'c-eldoc nil t) (progn (c-eldoc-print-current-symbol-info)))) ((string= lang \"css\") (if (require 'css-eldoc nil t) (progn (css-eldoc-function)))) ((string= lang \"php\") (if (require 'php-eldoc nil t) (progn (php-eldoc-function)))) ((or (string= lang \"go\") (string= lang \"golang\")) (if (require 'go-eldoc nil t) (progn (go-eldoc--documentation-function)))) (t (let ((doc-fun (org-eldoc-get-mode-local-documentation-function lang)) (callback (car args))) (if (functionp doc-fun) (progn (if ... ... ...))))))))
  org-eldoc-documentation-function(#f(compiled-function (string &rest plist) #))
  run-hook-with-args-until-success(org-eldoc-documentation-function #f(compiled-function (string &rest plist) #))
  eldoc-documentation-default()
  eldoc--invoke-strategy(nil)
  eldoc-print-current-symbol-info()
  #f(compiled-function () #)()
  apply(#f(compiled-function () #) nil)
  timer-event-handler([t 0 0 500000 nil #f(compiled-function () #) nil idle 0 nil])
" Disable showing Disable logging