Complete the initial LSH implementation

[lhecker]

Previous commits have added all of the infinity stones.
This one introduces the remaining, more complex .lsh files.

Closes #18

[MamiyaOtaru]

looking forward to the other languages. they seemed to work well enough in https://github.com/microsoft/edit/tree/4f36e2afe2a84ed339829ea6c63242fb9f4b7de3 (well, really long Here Strings made the rest of the file red, but ones with a reasonable length were fine)

[trumblejoe]

PR awaiting 2 pending checks still.

[lhecker]

Working on this feature? In the current economy?!
(I'll try...)

[Consolatis]

Thanks for your work on this. I've played around with this branch a bit and want to share my thoughts on the current implementation.

I tried to hack together basic C highlighting and noticed there is some state missing for "constants" (convention with all-uppercase variables usually defined with #define or part of some enum, not usual variables defined with const). Those are neither a Keyword nor a Variable.

Could the regexes be loaded from a file in the future? It might save some work to implement the basic functionality now by parsing a hardcoded string rather than defining everything in build/lsh/definitions.rs in rust code. Then it just needs the decision of what file(s) and from where to load + implement the actual file loading at a later point.

Another thing I wondered about is why the HighlightKind enum contains any colors, I have a feeling that those should be translated by themes from states like Comment or String.

Edit:
Some more feedback: support for Look::WordAscii (\b) and/or Look::WordStartAscii (\<) + Look::WordEndAscii (\>) would be very useful. Currently I can differentiate between break and breakage but not between break and unbreak. I've tried to implement it myself in build/lsh/compiler.rs but I am not familiar enough with rust and the regex-syntax crate.

[lhecker]

FYI I've continued development in the https://github.com/microsoft/edit/tree/dev/lhecker/syntax-highlighting-alt branch where I've since written a custom language with proper compiler. It even has rudimentary support for variables. It's fairly time consuming to develop so it's taking a while. It's also certainly not done yet, but please feel free to check it out already. You can find the definition files under crates/lsh/definitions.

[MamiyaOtaru]

nice to have PS highlighting back! quick comparison with highlighting as it was in the version I linked above

unrelated to highlighting: mousescroll worked in file dialog in the old one, scrolls edit window instead in the current syntax-highlighting-alt

[DanPartelly]

Why not tree-sitter integration for syntax highlighting and text objects ? You basically get for "free" countless languages. (implementation not considered)

[DHowett]

Why not tree-sitter

FWIW, this has been hashed out over in #18. In fact, the justification for having a simple syntax highlighter that is expressly not tree-sitter is described in the issue body of #18.

TL;DR (and it really is not much longer than this comment here): tree-sitter is huge. Edit is 250KiB. We don't want edit to become much larger than it already is.

[lhecker]

Well, so this ain't a "draft for syntax highlighting" anymore by any measure. This PR is now 7000 lines and contains an entire frigging compiler lol. (...which btw is extremely poorly engineered. 😭😭)

To anyone who reads this: Have fun testing this! Definitions are in crates/lsh/definitions.

[1001encore]

Hi, I've been experimenting with this and managed to get a basic markdown.lsh working with no lazy matching, since the engine only supports greedy matching. Successfully compiles to ~200 instructions. I thought it might be useful so posting below.
It doesn't render yet since the runtime seems to be work in progress, but I verified it via a CLI harness around lsh::compiler.

#[display_name = "Markdown"]
#[path = "**/*.md"]
#[path = "**/*.markdown"]
pub fn markdown() {
    // Headers
    if /#+ .*/ {
        yield keyword;
    }

    // Bold
    if /\*\*[^*]*\*\*/ {
        yield string;
    }

    // Links
    if /\[[^\]]*\]\([^)]*\)/ {
        yield function;
    }
}

[lhecker]

I'm glad you had a chance to try it out! I'll try to simplify the syntax in the future.

Did you try my markdown.lsh version? It's included in this PR: https://github.com/microsoft/edit/blob/cb98cf5b0fe6a5e192f37e82ef3716cd9a63db7f/crates/lsh/definitions/markdown.lsh

[1001encore]

oh, I totally missed that, my bad. I was on the compiler branch the whole time!

[DHowett]

15/19

[DHowett]

in JSON would this be an error token?

[DHowett]

do we have a token type for "this is obviously incorrect"

[lhecker]

Not yet, no. I didn't add that, because VS Code doesn't do anything special either. In an ideal world, Edit would get LSP support in the future which would significantly improve this.

[DHowett]

make sure you update the description!

[DHowett]

The powershell one seems somewhat broken, and yaml has some minor bugs, but I think this is a good place to start iterating.

[avih]

Not familiar with "LSH", and I'm not building it myself, so I didn't try it out.

Could it please be clarified whether the syntax schemes and/or the color themes are embedded into the binary? or as external files?

And if external files, where could a collection of such files be found?

Thanks.

[lhecker]

First off: This is the first (!) version of the simple (!) syntax highlighter. You cannot expect this to be feature complete.
(Edit: It may be worth mentioning that someone deleted their comment above, so my response may seem out of place.)

Alright...

Could it please be clarified whether the syntax schemes and/or the color themes are embedded into the binary? or as external files?

They're embedded and not configurable. I will release v2.0.0 without it being configurable, because I expect v3.0.0 to be all about configurability anyway.

And if external files, where could a collection of such files be found?

Hence, no external files. It's still a single binary.

I dont see any documentation in the commit - does adding a new .lsh require a recompile?

Yes. The source code gets translated to the VM bytecode at compile time. The compiler is fairly poorly written for my standards, and I do not want it to process possibly untrusted, dangerous .lsh files.

[lhecker]

@3052 If you'd like to further discuss this, create a "Discussion" and I'll respond there. We maintainers will not respond on this issue anymore.

And because I am the primary maintainer, I'll also take my right to collapse all prior comments above, because I'm trying to focus on drafting the preview release and this is distracting.