import os
import shutil
import tempfile
from fastcore.test import *
from trouver.helper.tests import _test_directory
markdown.obisidian.personal.notation
Add notation notes to Notation index note
notations_to_add_in_index
notations_to_add_in_index (vault:os.PathLike, notation_index_note=<class 'trouver.markdown.obsidian.vault.VaultNote'>, subdirectory:Optional[os.PathLike]=None, note: Optional[trouver.markdown.obsidian.vault.Vault Note]=None)
*Returns notations and links of notation notes to that ought to be added in the corresponding notation index, i.e. are in the reference folder but not linked by the notation index note.
If a notation note is not properly formatted, e.g. does not have a notation, then the notation and link for the notation note will not be included.
Raises - ValueError - If subdirectory
and note
are both None
.*
Type | Default | Details | |
---|---|---|---|
vault | PathLike | Path to the vault directory. | |
notation_index_note | type | VaultNote | The notation index note in the vault where the notations should be added to. |
subdirectory | Optional | None | Path to the subdirectory, relative to vault , to find the notation notes. Searches for all notation notes here and in subdirectories of this subdirectory. If None , then the note parameter is used to determined the subdirectory. If subdirectoryis the empty str, then all notation notes in the vault are searched. Defaults to None. | | note | Optional | None | The directory that this note is in determines the argument to subdirectoryparameter if it is None. Defaults to None, in which case subdirectorymust be specified. | | **Returns** | **list** | | **Each tuple in the list consists of the notation str of the notation note (including surrounding dollar signs $`) and the (nonembedded) ObsidianLink object for a link to the notation note.** |
We can identify notation notes which exist in a reference folder but are not linked in the notation index note for the reference:
# TODO: make test
# VaultNote.clear_cache()
= _test_directory() / 'test_vault_6'
vault = 'number_theory_reference_1'
reference = VaultNote(vault, name=f'_index_{reference}')
note = VaultNote(vault, name=f'_notation_{reference}')
notation_index_note = notations_to_add_in_index(vault, notation_index_note, note=note)
sample_output for notation_str, link in sample_output:
print(notation_str, link.to_string())
$\mathbb{Z}/n\mathbb{Z}$ ![[number_theory_reference_1_notation_Z_nZ_ring_of_integers_modulo_n]]
index_notation_note_formatted_entry
index_notation_note_formatted_entry (notation_str:str, link:trouver.markdown.obsidian.links .ObsidianLink)
*Return a str formatted for an index notation note entry.
It is recommended to pass the outputs of notations_to_add_in_index
to this function.*
Type | Details | |
---|---|---|
notation_str | str | The str of the notation, including the surrounding dollar signs $ . |
link | ObsidianLink | The embedded link to the notation note. |
Returns | str |
The index_notation_note_formatted_entry
function returns a formatted str to add in the index notation note:
print(index_notation_note_formatted_entry(sample_output[0][0], sample_output[0][1]))
### $\mathbb{Z}/n\mathbb{Z}$
- ![[number_theory_reference_1_notation_Z_nZ_ring_of_integers_modulo_n]]
Making a notation note
make_a_notation_note
make_a_notation_note (main_note:trouver.markdown.obsidian.vault.VaultNot e, vault:os.PathLike, notation:str, description:str, notation_note_name:str, destination:Optional[os.PathLike]=None, overwrite:bool=False, add_to_main:bool=True, latex_in_original:str='')
*Make a new notation note, optionally add a link to it in the See Also
section of its main note, returns it.
The notation note is created in the same directory as the main note. The meta of the notation note has a latex_in_original
section which lists the contents of the latex string in the main note from which the notation note comes from. This is so that the make_notation_notes_from_double_asts
method can distinguish between notations for which a note has been created and for which a note has not been created.*
Type | Default | Details | |
---|---|---|---|
main_note | VaultNote | The note from which the notation originates. | |
vault | PathLike | ||
notation | str | The notation typed in latex. May or may not be surrounded by dollar signs | |
description | str | The rest of the text describing notation. | |
notation_note_name | str | The name of the new notation note to be created. | |
destination | Optional | None | The directory to create the new notation note in. If None , then creates the new notation note in the same place as the note specified by note_name |
overwrite | bool | False | If True , overwrite file of the same path as the new notation file to be written, if such a file exists. Otherwise, does nothing. Even if a link to the old notation note exists in main_note , a new link will still be added. Defaults to False . |
add_to_main | bool | True | If True , adds a link to the notation note in the See Also section of the main note. |
latex_in_original | str | The full math mode string in main_note which introduces the notation. Defaults to the blank string '' , in which case notation plays the role of latex_in_original |
|
Returns | Optional | The newly created notation note. If no note is created, then returns None . |
We can make a notation note with the make_a_notation_note
method.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='galois_group')
note = make_a_notation_note(
notation_note r'\operatorname{Gal}(L/K)', '', notation_note_name='some_reference_name_notation_Gal_L_K_galois_group')
note, temp_vault, = MarkdownFile.from_vault_note(notation_note)
mf assert mf.has_metadata()
= mf.metadata()
meta assert 'detect_regex' in meta
assert 'latex_in_original' in meta
print(meta, '\n')
assert '\\operatorname{Gal}(L/K)' in meta['latex_in_original']
print(mf, '\n')
= MarkdownFile.from_vault_note(note)
main_mf # print(main_mf)
assert notation_note.name in str(main_mf) # A link has been created
# os.startfile(temp_vault)
# input()
{'detect_regex': [], 'latex_in_original': ['\\operatorname{Gal}(L/K)'], 'tags': []}
---
detect_regex: []
latex_in_original: ["\\operatorname{Gal}(L/K)"]
tags: []
---
$\operatorname{Gal}(L/K)$ [[galois_group|denotes]]
Note that the surrounding dollar signs for LaTeX math mode can be included in the argument for notation
:
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='galois_group')
note = make_a_notation_note(
notation_note r'$\operatorname{Gal}(L/K)$', '', notation_note_name='some_reference_name_notation_Gal_L_K_galois_group')
note, temp_vault, = MarkdownFile.from_vault_note(notation_note)
mf assert mf.has_metadata()
= mf.metadata()
meta assert 'detect_regex' in meta
assert 'latex_in_original' in meta
print(meta, '\n')
assert '\\operatorname{Gal}(L/K)' in meta['latex_in_original']
print(mf, '\n')
= MarkdownFile.from_vault_note(note)
main_mf # print(main_mf)
assert notation_note.name in str(main_mf) # A link has been created
# os.startfile(temp_vault)
# input()
{'detect_regex': [], 'latex_in_original': ['\\operatorname{Gal}(L/K)'], 'tags': []}
---
detect_regex: []
latex_in_original: ["\\operatorname{Gal}(L/K)"]
tags: []
---
$\operatorname{Gal}(L/K)$ [[galois_group|denotes]]
Setting add_to_main=False
only creates the notation note, but does not add a link to the notation note in the main note:
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='galois_group')
note = make_a_notation_note(
notation_note r'\operatorname{Gal}(L/K)', '', notation_note_name='_reference_notation_Gal_L_K_galois_group',
note, temp_vault, =False)
add_to_main
= MarkdownFile.from_vault_note(notation_note)
mf assert mf.has_metadata()
= mf.metadata()
meta assert 'detect_regex' in meta
assert 'latex_in_original' in meta
print(meta, '\n')
assert '\\operatorname{Gal}(L/K)' in meta['latex_in_original']
print(mf, '\n')
= MarkdownFile.from_vault_note(note)
main_mf # print(main_mf)
assert notation_note.name not in str(main_mf) # No link has been created
{'detect_regex': [], 'latex_in_original': ['\\operatorname{Gal}(L/K)'], 'tags': []}
---
detect_regex: []
latex_in_original: ["\\operatorname{Gal}(L/K)"]
tags: []
---
$\operatorname{Gal}(L/K)$ [[galois_group|denotes]]
If the notation note of the specified name (notation_note_name
) already exists, then by default no note is created and no link is added in the main note.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='spectrum_of_a_ring')
note = make_a_notation_note(
notation_note r'\operatorname{Spec} A', '', notation_note_name='some_reference_name_notation_Spec_A')
note, temp_vault,
assert notation_note is None
= MarkdownFile.from_vault_note(note)
main_mf # print(main_mf)
assert main_mf.get_headings_and_text()['# See Also'].strip() == '' # No link has been added
Setting overwrite=True
, however, will overwrite the existing note. The method will also add a link to the (overwritten) notation note.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='spectrum_of_a_ring')
note = make_a_notation_note(
notation_note r'\operatorname{Spec} A', '', notation_note_name='some_reference_name_notation_Spec_A',
note, temp_vault, =True)
overwrite
assert notation_note is not None
= MarkdownFile.from_vault_note(note)
main_mf assert notation_note.name in str(main_mf)
= MarkdownFile.from_vault_note(notation_note)
notation_mf # notation_mf has been overwritten
print(notation_mf)
---
detect_regex: []
latex_in_original: ["\\operatorname{Spec} A"]
tags: []
---
$\operatorname{Spec} A$ [[spectrum_of_a_ring|denotes]]
Make notation notes from double asterisks surrounding notations
make_notation_notes_from_double_asts
make_notation_notes_from_double_asts (main_note:trouver.markdown.obsidia n.vault.VaultNote, vault:os.PathLike, reference_name:str, destination:Opt ional[os.PathLike]=None, overwrite:bool=False, add_to_main:bool=True)
*Make notation notes based on double asterisks surrounding LaTeX text in a standard information note.
Notations are deemed to be completely LaTeX text in info notes that are surrounded by double asterisks. In basicality, if such a LaTeX text (without surrounding dollars signs $
or $$
) is listed in the latex_in_original
metadata section of some notation note in the same directory as the info note whose main note is the info note in question, then a new notation note for that LaTeX text is not created. However, if there are multiple instances of the same LaTeX text, then some notation notes may be created so that the number of times the LaTeX text appears in the info note is the no more than the number of times the LaTeX text appears in latex_in_original
metadata sections of notation notes (in the same directory as the info note whose main note is the info note).
For example, if there is an info note with notations A
, A
, 'A'
, 'A'
, and B
and if there is a single notation note in the same directory as the info note with two 'A'
and 'A'
entries in its latex_in_original
metadata section, then three notation notes will be created: two with 'A'
listed in their latex_in_original
sections, and one with 'B'
listed in its latex_in_original
section.
Raises
- Warning
- If there are notation notes whose main note is determined to be to
main_note
and whose notations “excessively cover” those inmain_note
, i.e. the notation notes have more notations thanmain_note
introduces. The main note and the excessive notations are printed; the notations are printed instead of the notation notes because the same notation may span either multiple or single notation notes.*
- If there are notation notes whose main note is determined to be to
Type | Default | Details | |
---|---|---|---|
main_note | VaultNote | The standard information note from which the notations are marked with double asterisks | |
vault | PathLike | The name of the reference; the notation note’s name will start with {reference_name}_notation_ . |
|
reference_name | str | ||
destination | Optional | None | The directory to create the new notation notes in. If None , then creates the new notation note in the same place as the note specified by note_name |
overwrite | bool | False | If True , overwrite file of the same path as the new notation file to be written, if such a file exists. Otherwise, does nothing. Defaults to False . |
add_to_main | bool | True | If True , adds links to the notation note in the See Also section of the main note. |
Returns | list | The list of VaultNotes that are newly created/modified. |
As described in markdown.obsidian.personal.machine_learning.notation_identification
, we surround a LaTeX math mode string with double asterisks **
to indicate that the string introduces a notation.
The make_notation_notes_from_double_asts
method parses LaTeX surrounded by double asterisks **
in a standard information note and automatically creates notation notes for said LaTeX.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_6'
temp_vault / 'test_vault_6', temp_vault)
shutil.copytree(_test_directory()
# os.startfile(temp_vault)
# input()
= VaultNote(temp_vault, name='reference_for_notation_notes_introducing_some_notations')
info_note = make_notation_notes_from_double_asts(info_note, temp_vault, 'reference_for_notation_notes')
new_notes
assert len(new_notes) == 3
for new_note in new_notes:
assert new_note.exists()
assert notation_note_is_linked_in_see_also_section(new_note, info_note)
# input()
# TODO: add more tests - overwrite=True, add_to_main=False
In the following example, we prompt make_notation_notes_from_double_asts
to make notation notes for an info note with no notations - nothing is modified:
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_6'
temp_vault / 'test_vault_6', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='reference_for_notation_notes_no_notations_introduced_here')
info_note = info_note.text()
info_note_content_before = make_notation_notes_from_double_asts(info_note, temp_vault, 'reference_for_notation_notes')
new_notes = info_note.text()
info_note_content_after
assert len(new_notes) == 0
assert info_note_content_before == info_note_content_after
In the following example, we prompt make_notation_notes_from_double_asts
on the same info note twice - no new notation notes are created the second time.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_6'
temp_vault / 'test_vault_6', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='reference_for_notation_notes_introducing_some_notations')
info_note 'milne_av')
make_notation_notes_from_double_asts(info_note, temp_vault, = make_notation_notes_from_double_asts(info_note, temp_vault, 'milne_av')
new_notes assert len(new_notes) == 0
In the following example, an info note contains two of the same notation. One notation note for each of these notations is created, but with different names:
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='note_with_repeated_notation')
info_note = make_notation_notes_from_double_asts(info_note, temp_vault, 'some_reference_name')
new_notes assert len(new_notes) == 2
print(new_notes[0].name)
print(new_notes[1].name)
some_reference_name_notation_Cl_K
some_reference_name_notation_Cl_K_1
In the following example, there are notation notes with the info note as their main note, but some of the notations in these notation notes cover those in the info note “excessively” - in this case, only notation notes to uncovered notations are created, and warnings are raised to indicate which notations are covered excessively.
with tempfile.TemporaryDirectory(prefix='tmp_dir_', dir=os.getcwd()) as tmp_dir:
= Path(tmp_dir)
tmp_dir = tmp_dir / 'test_vault_7'
temp_vault / 'test_vault_7', temp_vault)
shutil.copytree(_test_directory()
= VaultNote(temp_vault, name='note_with_some_excessive_notation_notes')
info_note = make_notation_notes_from_double_asts(info_note, temp_vault, 'some_reference_name')
new_notes assert len(new_notes) == 1
C:\Users\hyunj\AppData\Local\Temp\ipykernel_23928\60597109.py:57: UserWarning: The following note has the following excess notations: note_with_some_excessive_notation_notes, \mathcal{B}(\mathbb{R}), \operatorname{Jac}(C)
warnings.warn(
# TODO: move notation notes to directory of main notes
Deleting notation notes
Recall that we ue HTML tags to mark definitions and notations, cf. markdown.obsidian.personal.machine_learning.tokenize.def_and_notat_token_classification
. We also use markdown.obsidian.personal.machine_learning.definition_and_notation_naming
to generate “names” to definition and notation tags. In particular, the “names” for the notation tags will be the notation str for the notation note, i.e. the notation note will be of the format
[link_to_information_note|denotes]] ... $<notation str>$ [
In practice, we might 1. apply the functions in markdown.obsidian.personal.machine_learning.tokenize.def_and_notat_token_classification
, 2. apply the functions in markdown.obsidian.personal.machine_learning.definition_and_notation_naming
, and 3. apply the make_notation_notes_from_HTML_tags
function to create notation notes. However, steps 1 and 2 are performed by ML models and hence can often be incorrect.
While it is inevitable to manually fix the errors generated in steps 1 and 2, it is a hassle to manually fix the corresponding notation notes — doing so is effectively trying to fix the same mistake twice. Instead, we implement the below functions to recognize notation notes that are autogenerated mistakenly and remove/replace them as necessary.
notation_note_has_no_verified_content
notation_note_has_no_verified_content (notation_note:trouver.markdown.ob sidian.vault.VaultNote)
Return True
if notation_note
has no genuine content or if the content is determined to have been auto-generated. is determined to not have any genuine content or the content is auto-generated.
remove_bad_notation_notes
remove_bad_notation_notes (main_note:trouver.markdown.obsidian.vault.Vau ltNote, vault:os.PathLike, links_in_vault:Opti onal[dict[str,list[str]]]=None)
*Remove “bad” notation notes associated to main_note
A “bad” notation note is one which satisfies all of the following:
- is determined to essentially have no verified content (via the
notation_note_has_no_verified_content
function). - is not linked to anything in
vault
except formain_note
. # 3. all entries of thelatex_in_original
field in the YAML frontmatter meta are not present inmain_note
*
Type | Default | Details | |
---|---|---|---|
main_note | VaultNote | The standard information note in which the notations are marked with HTML tags and which notation notes are to be removed as appropriate. | |
vault | PathLike | ||
links_in_vault | Optional | None | An output to all_links_in_vault with backlinks set to True . If None , then this is computed on-the-fly. |
Returns | list | The list of VaultNotes that are newly created/modified. |
# TODO example
reorder_notation_note_links_in_see_also_section
reorder_notation_note_links_in_see_also_section (main_note:trouver.markd own.obsidian.vault.Vault Note, vault:os.PathLike)
*Reorder the bulleted links to the notation notes in the # See Also
section to match the order of the “latex in original” latex strings as they appear in main_note
.
Assumes that the # See Also
section begins with bulleted links to notation notes and that all bulleted links to notation notes in main_note
reside at the beginning of the # See Also
section.*
# TODO: example
Decomposition of notation
# #| export
# def decompose_notation_as_sequence(
# notation # latex styled. Assumed to not be surrounded with `'$'`.
# ) -> list:
# """
# **Parameters**
# - notation - str
# **Returns**
# - list of str
# """
# str_index = 0
# decomposition = []
# while str_index < len(notation):
# if notation[str_index] == '\\':
# j = 1
# while (str_index + j < len(notation)
# and notation[str_index + j] not in [' ', '\\', '{', '(']):
# j += 1
# latex_command = notation[str_index:str_index + j]
# decomposition.append(latex_command)
# str_index += j
# if str_index < len(notation):
# decomposition.append(notation[str_index])
# str_index += 1
# return decomposition
# def compare_notations_for_sorting(
# notation1, # latex styled. Assumed to not be surrounded with `'$'`.
# notation2, # latex styled. Assumed to not be surrounded with `'$'`.
# character_ordering_list
# ) -> int:
# """
# **Parameters**
# - notation - str
# - latex styled. Assumed to not be covered with `'$'`.
# **Returns**
# - int
# - 1 if `notation2` is considered to come "earlier", -1 if
# `notation1` is considered to come "earlier", and 0 otherwise.
# """
# decomposition1 = decompose_notation_as_sequence(notation1)
# decomposition2 = decompose_notation_as_sequence(notation2)
# index1, index2 = 0, 0
# while index1 < len(decomposition1) and index2 < len(decomposition2):
# return #TODO
# if index1 < len(decomposition1):
# return 1
# if index2 < len(decomposition2):
# return -1
# return 0
# def _find_next_effective_character(decomposition, index):
# while (index < len(decomposition)
# and decomposition[index] in ['{', '}', '(', ')', '[', ']', r'\tilde',
# r'\hat', r'\bar', 'r\overline']):
# return
# decompose_notation_as_sequence(r'\tilde{K} (X)')
# decompose_notation_as_sequence(r'P(X;*,A)')
# decompose_notation_as_sequence(r'\operatorname{Spec} A')
Detect notations being used in reference
Regex from latex
regex_from_latex
regex_from_latex (latex:str, replaceables:dict[str,set[str]]={'mathrm': {'rm', 'operatorname', 'text', 'mathrm'}, 'operatorname': {'rm', 'operatorname', 'text', 'mathrm'}, 'rm': {'rm', 'operatorname', 'text', 'mathrm'}, 'text': {'rm', 'operatorname', 'text', 'mathrm'}, 'mathbf': {'bf', 'mathbf'}, 'bf': {'bf', 'mathbf'}, 'mathit': {'mathit', 'it'}, 'it': {'mathit', 'it'}}, special_characters:list[str]=['.', '+', '*', '?', '^', '$', '(', ')', '[', ']', '{', '}', '|', '\\'])
*Returns regex to match latex math mode string which is essentially equivalent to a specified latex math mode string.
The outputs of this function may not work correctly. The regex pattern does not have to fully match equivalent string.
Parameters
- latex - str
- The latex math mode string. Does not include math mode delimiters such as
$
,$$
,\[ \]
(although the characters'\['
and'\]'
can still be part of the string, e.g. for optional arguments of a macro/operator). Can include “placeholders”r'\1'
,r'\2'
,r'\3'
, etc. to indicate substitutable/generics; the placeholders can be substituted with any string.
- The latex math mode string. Does not include math mode delimiters such as
- replaceables - dict[str, set[str]]
- latex strings/commands which are considered “interreplacable”
- special_characters - list[str]
- characters to add a backslash
'\'
in front of for regex. Defaults to a list consisting of special characters in regex.*
- characters to add a backslash
= r"""e"""
text print(regex_from_latex(text, REPLACEABLES))
(?:[ \{\}]*?)e(?:[ \{\}]*?)(?:[ \{\}]*)
Get regex from notation note
So far, I have just made notation notes in the form '$math_mode_string$ denotes ...'
. I want to add frontmatter metadata in notation notes to indicate regex to detect the notation with placeholders.
regex_from_notation_note
regex_from_notation_note (vault:os.PathLike, note:trouver.markdown.obsidian.vault.VaultNote)
*Returns a regex str to detect the notation of the notation note.
The regex detection strings should be in a list labeled detect_regex
in the yaml frontmatter. If multiple strings are in the list, then the regex will detect latex math mode strings roughly corresponding to any of them. If multiple strings are in the list, then they must be ordered “by priority”, with the higher priority regexes coming first. It is good to have these string in quotes ""
to make sure that yaml can load them safely. When doing so, make sure to escape characters, e.g. backslash should be typed as \
, etc.
The strings in detect_regex
can include placeholders, cf. regex_from_latex
.
Parameters - vault - PathLike - note - VaultNote
Returns - str - Of the regex used to detect the notation. The regex does not need to fully match instances of the notation.*
# TODO: test