/tech/ - Technology

>>34069
Rewrote the lexer to pull out a PDA with finite registers.
The caller provides parsers of a regular languages with state transitions, and fallbacks.
They do this by providing a large or'd sequence of named regex groups.
They then call the transition on the returned match.lastgroup, and supplied start, end pair.
Or if there is no match on the gap between the last position and the next match or EOF.
This should allow for the two phase lexing of commonmark as well as better handling of some blocks.

Next up is to generalize the parser to a LL(1) framework.
The caller will provide a REGISTRY table containing functions Mode -> Token -> MutableState -> MutableState
Where the MutableState is mostly just a Stack of pointers into the accumulator, an accumulator and a return.

The tricky part of course will be to actually write out all the lex and parse rules for commonmark with extensions.
On this note found the following interesting program which will serve as the basis of rewriting my rules:
https://github.com/mmark-md/mmark/blob/master/Text/MMark/Parser.hs

Have been learning a little by doing…
The regex engine I was using doesn't support unicode ranges over byte data.
Meaning it doesn't support unicode ranges over mmap data.
So I have to rip it out and replace it with hyperscan/chimera which will be faster.
Also realized I wanted the option to support scan and match modes.
And a stack of transition tables instead of transition functions to allow for extensions more easily.

Additionally attempts to implement the inline code already blew up the parser.
It now quite clearly has to be LL(k),

have you tried mistune?

Maybe it's because i just have a fundamental dislike for markdown and generating everything downstream from an mediocre markup format, but i think you may be approaching this problem the wrong way.

I recently needed a spaced repetition system and what did i do? I implemented a pseudo-VFS with Common Lisp structs and made it interface with the filesystem using "blob" and "dot" types to track regular files and files containing knowledge representation objects respectively. Dots are either read-only and loaded as a type of module or lisp objects which can be printed back and i've settled on doing everything with (potentially nested) plists and a path search struct. Finally the core program loads a lisp rc file and execs into a custom command-line with its own restarts and a cwd.

Then i wrote a couple of modules that define

• auto-promotion of blobs with an .a extension to writable dots
  
• multi-language keywords for storing information in dots
  
• a field for links to other dots and a function for plotting a path (which i call a trace) based on them
  
• a testing command which goes through every dot in and connected to a prefix as a topical graph ordered randomly and by the order of links and asks questions according to a plan like ('name summary :equation)
  
• commands that further extend the notion of a "trace" in order to pass as a memex
  
  

Since i'm still in the process of writing the emacs integration, the interface is currently quite rudimentary, yet i'm happy with what it does regardless.

I wrote the lexer as described in >>34069 but it turns out callbacks to python every match is too slow.
Like three orders of magnitude too slow…
It's very customizable which is nice, the API would not be impossible in C without manual patching.
Mostly because of the protocols which allow for the Context and State to be extended.
It's available at: https://codeberg.org/jugaad/lexify

>>34160
No, but I was having too much fun to try something else!

>>34161
Wrote a VFS once on top of indexeddb had a comprehensive enough interface to run isomorphic-git, and a complete test suite.
Wanted a MDX-editor with git implemented as PWA, but of course it has to have an FS menu so…
Should have just bought the bullet and gone native, might have had a mini product if I had.
Funny how everything around here is so…