/tech/ - Technology

>>30804
i'm going to guess "for employability reasons", i think yugoanon is choosing technologies based on his career path

>>30805
python is highly in demand though?

>>30804
>>30805
>>30821
Ended up taking a break from this project, and school and work are about to kick in so it's probably for the best.
The reasons for picking Typescript are as follows:
1. https://fedify.dev/ is way better than the python alternatives.
2. Wanted to be able to apply to more jobs, was thinking it was a question of quantity (probably wrong).
(The fresh Island architecture and preact also seems nice.)

Seems like my contact fell through so am back to programming. arsvia-redux has expanded and transformed dramatically in scope - as vaporware. The idea now being a social knowledge base with some AI integration, and anonymous content. Meanwhile an arsvia rewrite seems underway using some of the lessons from >>30995 but hilariously in a bottom up, test later manner. Might actually make a hosted instance for dogfooding purposes this time. Truthfully I'm much more interested in the much more difficult redux project, but this is still good practice, and it's almost outlandish.

Why tho? Normal Vichan gets updates sometimes.
I wish the container stuff worked better.

>>31197
>Why tho?
arsvia-redux wouldn't have an equivalent, it might even turn a profit. The version two however is hard to justify. Seems like just a test bed for some ideas, namely having a large domain model with a "separation" between functional core and imperative shell. Maybe some stuff with testing (later). Also wouldn't mind a 4chan compatible API, and cache invalidation, but there's no real reason besides wanting it to work better.

>>31199
Have already run into issues with what really should work as far as SQLAlchemy validators go. The following seems impossible:

@validates("thread")
async def _validate_thread_in_db (self, thread: int) -> None:
    thread_post = await self._session.execute(
        select(Post)
        .where(Post.id == thread)
        .where(Post.thread == thread)
        .where(Post.status != PostStatus.DELETE)
    )
    if thread_post is None:
        raise ValueError("thread no longer exists, or never did.")

The issue is that it can't validate async functions. Might try again tomorrow or soon with SQLModel which supposedly can manage this sort of thing.

>>31200
>SQLModel which supposedly can manage this sort of thing
The closest solution found was here [^1], but it incompatible with SQLModel.
Am considering using Pydantic and aiosql [^2] instead, just for this.
Sort of postmodern in a good way, and imagine it would require fewer hacks.
That is if have the energy to do non-trivial refactoring in this rewrite.

:[^1] https://github.com/team23/pydantic-async-validation
:[^2] https://nackjicholson.github.io/aiosql/

>>31199
Well as the only one here who runs an imageboard I volunteer netherchan for testing.

>>31203
This is very kind of you James.
There wouldn't be much of a reason to run this version two arsiva.
Your styles wouldn't apply, and there's no migration scripts yet.
Even if it all worked you just wouldn't be getting much beyond vichan.
But arsvia-redux could be interesting if you wanted to restart and I had the energy.
Main problem is it seems like it would just be a ton of work, and it's still in planning.

>>31200
>>31202
Think the basic problem with this validator is that its relational, but in the class.
It's the same problem with the handling of bumping and locking of full threads.
There's still no way to slice through the circularity of File and Post validation.
Closest seems to be to have a single insert and validate function.
Hints at a larger PostWithFiles class.

>>31208
The following seems to work well.
Unfortunate that resolve_references is still in the init function.
This makes the functional core a little smaller.

class Post (BaseModel):
    metadata: PostMetaData
    references: List[Reference] = []
    files: List[File] = []
    comment: str = ""

    @model_validator(mode="after")
    def _validate_files (self) -> Self:
        if self.metadata.thread == self.metadata.id and len(self.files) == 0:
            raise ValueError("zero files provided, expects at least one.")

        if len(self.files) > config.BOARDS[self.metadata.board]['MAXIMUM_FILE_COUNT']:
            raise ValueError("post exceeds the maximum file count.")

        return self

    def __init__ (
        self,
        connection,
        board: str,
        thread: Optional[int],
        user: Optional[UUID],
        address: UUID,
        subject: Optional[str],
        file_storages: List[FileStorage],
        spoiler: bool
    ):
        # XXX: Until insert metadata lacks the thread, and id.
        self.metadata = PostMetaData(board, thread, user, address, subject)

        for file_storage in file_storages:
            self.files.append(File(self.metadata, file_storage, spoiler))

        parse_tree = parse_with_partial_references(self.metadata.board, self.comment)
        # XXX: We must query for the thread of posts referenced to form link.
        parse_tree, self.references = resolve_references(parse_tree, connection)

        self.comment = render(parse_tree)

>>31212
Moving in the direction of witness objects containing the bulk of the validation, and lots of frozen classes with static methods, keeping the imperative as small as possible, and separated though not in separate modules. Key difference from the plan is starting (probably erroneously) with the types, an validation logic. That is start by structuring the state-space. There's also some "separation of concerns" between validation of the object, the object, and the persistence layer, whereas was planning on just having a massive single object for the domain model.

>>31216
Ended up with a pattern of creating a witness ObjectCreate class which handles validation of input. A ObjectRequest class which transforms in input into the object, and ObjectResponse for retrieving an object from the database, these then inherit from Object to share common field. I'm not exactly happy with it; this could have been done with two objects and a factory. The diagram doesn't contain the inheritance for some reason.

>>31221
Had to unfreeze all my dataclasses which is a bit annoying. Renamed the Object classes to be ObjectMixin which is a little more appropriate. Also was able to rewrite the schema today, the results in a viewer rendered attached. It's probably not going to be too difficult to get the queries to work too but building the Repositories may take a second. A sample of what (the simplest of) these look like follows:

def get_thread_post_responses (database: Database, board: str, thread: int) -> List[PostResponse]:
	posts, files, references = await asyncio.gather(
		PostRepository(database).get_replies_by_thread(board, thread),
		FileRepository(database).get_files_by_thread(board, thread),
		ReferenceRepository(database).get_back_references(board, thread)
	)

	post_files = {}
	for file in files:
		if post_files[file.post] is None:
			post_files[file.post] = [file]
		else:
			post_files[file.post] += [file]

	post_back_references = {}
	for reference in references:
		if post_back_references[references.referee] is None:
			post_back_references[reference.referee] = [reference]
		else:
			post_back_references[reference.referee] += [reference]
			
	post_responses = []
	for post in posts:
		post_response = PostResponse(
			**post,
			back_references=post_references[post.uuid],
			files=post_files[post.uuid]
		)
		post_responses.append(post_response)

	return post_responses

Sad.

>>31240
This is a good function, think it's going to do the bulk of the lifting for my unmarshaling to larger objects.

from functools import reduce
from typing import Any, Dict, List, TypeVar

T = TypeVar('T')
def group_by_field (xs: List[T], field_name: str) -> Dict[Any, List[T]]:
    def f (acc, x):
        k = getattr(x, field_name)
        return acc | {k: acc.get(k, []) + [x]}
    return reduce(f, xs, {})

>>31249
And yes optimization in python is apparently just using dictionaries in various ways.

>>31249
Imperativechads keep winning: https://perfpy.com/1057

def group_by_field2(xs: List[T], field_name: str) -> Dict[Any, List[T]]:
    result = {}
    for x in xs:
        key = getattr(x, field_name)
        result.setdefault(key, []).append(x)

    return result

>>31251
Dictionary union is apparently dog slow… https://perfpy.com/1058
I've switched to your version anyway, as it's also easier to understand.

>>31251
This seems even faster: https://perfpy.com/1059

def group_by_field3(xs: List[T], field_name: str) -> Dict[Any, List[T]]:
    result = {}
    for x in xs:
        key = getattr(x, field_name)
        if key in result:
            result[key].append(x)
        else:
            result[key] = [x]

    return result

>>31253
This doesn't even seem like it should be true! It's not that it's binding to the empty list conditionally that's slowing it down (it might even be a hair faster to do this https://perfpy.com/1060). Not sure what this thing could be doing, and why seemingly pretty basic abstractions would be slowing the thing down.

>>31256

result.setdefault(key, []).append(x)

Doing this in the hot loop means allocating an empty list in every iteration, even if `key' is already in result. It's probably why it is slower.

In case of #1060, there's an error for Code 2 and rerunning the whole thing a few times, it seems random which comes out as the winner. The memory is randomly changing between around 700 KB and 1.06 MB, something is slightly off.

>>31257
Apparently the best way forward is to avoid this fun entirely and just fetch using PostgreSQL native arrays. Then just need to convert the SoA to a AoS. This means rewriting several queries.

Ha! Victory! https://perfpy.com/1063

>>31261
No lambda works even better! https://perfpy.com/1064

>>31262
Think at this point I'm confident in all the classes but the reports, in the schema, and reasonably in the larger queries. Am in the process of replacing the smaller queries with bigger ones to output whole objects at a time. And still need to write the insertions. Won't be rewriting the front-end in preact as tempting as that is because users prefer not to need JavaScript. Of course redux would have something like this.

Is it strange that i'm considering having the frontend of my imageboard be a cgi script that calls into a special-purpose implementation of djb redo (a make replacement) to dynamically pick up the directory structure and plug the contents into a HTML template?

>>31267
Had an unscheduled break from this project today, hopefully it doesn't last.
Or otherwise get some momentum to switch to arsvia-redux (even better).
Would like 2ch.hk style recursive post preview.
This requires an additional endpoint to fetch individual posts.
It would also require JavaScript, which is making me reconsider.
Would also like to have the live-board feature be incremental to save on bandwidth.
Would almost be easier just to rewrite it in Fresh…

>>31272
A dynamic static site generator!

>>31273
>A dynamic static site generator!
Isn't that how /leftypol/ and basically all other imageboard work? They generate HTML files that are served statically.

>>31274
There's still a database no?

>>31274
The difference is the vichan is a ball of php and so are the templates. Separating representation from the data allows for arbitrary frontends, convenient modification of templates and seamless integration of plugins. This can be coupled with a directory cloning operation, that inherits build scripts or option files.

>>31274
Had a couple funny ideas: one about using HTTP Digest Authentication with passwords stored in /etc/shadow and an endpoint for registration which generated a OTP and sent it over email. Another was to use xattr for properties like file or reply count which wouldn't be as elegant in the file name.

In a different context was that it might be useful to store records like this for my redux application since archivability was one of the objectives. Just not sure about the format which would be desired for users to archive replies to their posts, or if that should be ignored entirely.

>>31277
>filename
Guess a custom fileformat makes this moot.

Found https://writefreely.org which is similar in many ways to my concept for redux. It seems like a good project and I'm glad it exists, but it's missing some of my features:

- OP moderated "threads"
- federated replies
- file versioning link-rot protection
- wiki-style links with user directories [[@user@instance/filename/version]]
- back-links
- distinction between zettlekasten and bibliographic notes
- AI "integrate knowledge" button.
- anonymous posting

This is a ton to implement so don't feel to anxious sharing on an esoteric Mongolian picture site. Would love to discuss here any additional features you'd like to see in such a application.

>>31282
Ugh, somehow forgot one:
- archivability of all content, and someway to archive replies to your content.

>>31282
I haven't looked into your projects, because i don't have a grasp on any of the frameworks or even the language. What it sounds like to me though, is that you're trying to write a kitchen sink chat/bbs/wiki/booru from the top down, which looks like a bad idea to me. First, if someone needs only one of these 4 main functions, they will rarely consider using your software over a more minimal and consequently simpler alternative. Second, forum software usually doesn't sit in a vaccuum. It exists to serve the need of a community and unusual functionality often accrues over time as the need arises.

When i write software, i try to minimize the set of core features while supporting as wide of a featureset as possible. This is the case for most programming language runtimes, which are consequently relatively easy to port even c/unix given the scope and also emacs. Your system may be robust, comprehensive and elegant, but to me it seems like an evolutionary dead end.

Soygeleno won btw.

>>31282
>>31283
so activitypub (+ some trick to obfuscate user identity) but with mentally retarded non-features: ai and hardcoded formatting

if it is federated and I want to setup an instance I have to also setup a local chatbot or pay a chatgpt commercial license? that's an absurd entry cost for what will be, at best, an extremely niche platform.

"link-rot protection" is always a meme because if I stop hosting something, it doesn't matter what the url was. if by link-rot you mean that content that is still hosted by the server becomes unreachable because of url changes, every single ib that stores files by their md5 hash (like 4chan) already has that feature. matter of fact, this is only a problem in enshittified corporate internet. do you also need to specify that your ib won't be vulnerable to script injection?

>>31284
>kitchen sink chat/bbs/wiki/booru
I think there is a scope, but it might be too big. Maybe the scope is rather a purpose, it is what it does. Anything which would improve my usage of my (and others') zettelkasten. [^1] (To put briefly something put much better by me in a different format today) I'd like to improve thinking.

>i try to minimize the set of core features while supporting as wide of a featureset as possible
Guess the challenge is in designing this… Am struggling to imagine what to get rid of!

>>31286
>hardcoded formatting
This is a very interesting critique to me. I don't think it would federate well if you had any options for formatting. You could certainly allow for customizable CSS and scripts though, just like a static site generator.

>if it is federated and I want to setup an instance I have to also setup a local chatbot or pay a chatgpt commercial license?
No, this was going to be an optional feature, disabled by default. :[^1] If there was some tool to help me finish books I've started taking notes on that would be even better, but this and the social impetus (of publishing) to do the right things, are the only things to mind that could encourage me to use a zettlekasten better.

>if I stop hosting something, it doesn't matter what the url was.
Haven't worked out the concept yet but it would be nice to have local archives of linked markdown pages (if this is recursive it could cause trouble). I believe foreign replies would be locally archived by default since they're sent to the inbox. And another idea was to send a reply event every time there's a markdown link for backlinks (but this would be impossible to do recursively).

>>31287
>Don't think it would federate well if you had any options for formatting.
This could be expanded to the Mastadon tag set which is reasonably large other than not including <div> or <table>…

>>31288
So if you make an Article instead of a Note you get unlimited length and HTML tags because you just link back to your instance in the feeds. Am reminded of my hatred for traditional WYSIWYG, and the following from six years ago is accessed in memory (pic rel): https://secretgeek.github.io/html_wysiwyg/html.html Bet could have an HTML editor that, like Obsidian LaTeX rendering, is rendered typically when the cursor is outside of them, and otherwise in this "brutalist" view. How hard could it be?

>>31291
Was playing around some more with this idea.
- Walk the tree
1. getting the tag name and attrs, setting :before and :after as needed to a generated class name.
2. setting tag decorator position: absolute and opacity: 0 by default
3. applying contenteditable=true to everything.
- When the cursor is inside or adjacent a tag display it. (only implemented currently on :hover)
- On insert create a new Node if that's necessary
- Some interface for editing existing tags???

This sort of thing gets a little hairy for more complicated websites.
Might be nice if the decorator was overlaid.
You could then have a toggle with perhaps a mode for all, or just the inside/adjacent.

Think the main trouble is that styling (mainly class) and boxing (div and span) are mixed in with the semantic tagging.
(The most trivial to remove is the contenteditable="true" which shouldn't have been there in the first place.)
Removing the attrs entirely by default is a massive improvement.
So yes, HTML is markup, you don't really need the translation layer, but it's also it's not really semantic or light.
On the other hand this sort of styling and boxing is only required for websites that are more complex than possible with markdown.
There could be an HTML to HTML static site generator in JavaScript for table of contents, MathJax, and similar.

>>31296
Before i tried wrangling HTML into an embeddable markup format, i would just bite the bullet and define my own markup dialect and write transpilers from a HTML subset, bbcode, etc.

Look, if you can't think of an elegant base framework for your features, i'll try doing the work for you:
An article contains a body (content), terminal tags (title, date, flag, uploaded files) and non-terminal tags (forum-style tags, thread number, thread number, tripcode). Non-terminal tags define relations between articles, allowing a client to submit posts to a thread on a given board or display all threads in bump order. Tripcodes act as a secure identifier and therefore allow user accounts or pages by assigning articles to an author or even submit a twitter-style thread under a disposable trip.

I think the above definition is concise enough to be a small subset of something like activitypub (the only "activities" would be submitting articles and retrieving articles), but general enough to implement everything you mentioned. User sessions, link walking and display would all be malleable, i feel like encoding all of this into a stripped-down format goes a long way though. Now it's your turn to build the next layer!

>>31298
To give you an idea about what this would look in practice, i will now define a few of your features on a very limited set of non-terminal tags: board, thread and tripcode:

The server is free to reject article submissions to boards or thread that have no previous representation or "don't exist" in whichever representation the server chose. The absence of a thread tag assigns the article a free thread id. Tripcodes act as user identifiers, therefore there can be a convention of showing the latest thread under the trip on /~users/ as a user page (allowing only one thread id per trip on this board and overwriting it on new submissions would make this more convenient). 'Recpients', a terminal field containing a list of tripcodes, would allow the server to discriminate who could retrieve articles, which would be necessary for implementing private messages (i would also dump these to a special purpose board).

>OP moderated "threads"
This one is tricky, but the most straightforward approach would be submitting to a moderation queue board and having the OP "resubmit" accepted articles.
>distinction between zettlekasten and bibliographic notes
Specify a terminal tag for each.

This scheme is fairly kludgy, but i think it gets the point across.

>>31298
The idea that's floating in my head (now) is to not embed HTML, but the application, as a set of optional widgets.
One widget displays backlinks as Article links, and replies as Notes, and allows for the creation of Notes (maybe three tags for these).
Another takes an inbox, outbox, or sharedInbox and creates a feed.
The base front-end for the application is a nearly full-screen modified contentEditable with a submit/edit button.
Authentication might be a little tricky… but should be possible…

I do think the distinction between relational data and formatting data is valuable.
Even though this is a smaller application, it still feels like it could be smaller.

>>31298
The parsing analogy is interesting. Think it's understood. Wonder if it generalizes?

>>31299
So this is about API design right?
You're saying there should be, in this case, a single endpoint.
With hidden fields in the form and the action attribute set.
The point is then to reduce the complexity of the transmitted data, endpoints, and render.
All while achieving a certain level of civilization.

This makes me want to get back to rewriting arsvia2, instead of idly contemplating redux.

>>31298
>>31299
Used this to delete a table, and simplified and renamed a few more. Similar effects on simplifying classes. Also consolidated the queries, so that there's really only two written now, but they service the tripcode, address (for admin), thread, catalog, and recent views. Think there might be three other pages so perhaps three more smaller queries to come. Think getting rid of the Mixins is the next big step for the rewrite.

Today only managed to get nested contentEdtable elements to work like they should.
The key idea is to zoom out the lens (all the way) for navigation and zoom in (all the way) for editing.
Focusing the body with addEditable is necessary for repeat keypresses.

<!DOCTYPE html>
<html>
<style>
*:focus { outline: none; }
</style>
<body>
<div id="outer">
some text
<div id="inner">
inner
</div>
more text
</div>
</body>
<script>
const body = document.body;
const outer = document.getElementById('outer');
const inner = document.getElementById('inner');

inner.addEventListener('mouseup', () => narrowToSelection(), true);
inner.addEventListener('touchend', () => narrowToSelection(), true);
outer.addEventListener('mouseup', () => narrowToSelection(), true);
outer.addEventListener('touchend', () => narrowToSelection(), true);

const isArrow = k => ['ArrowLeft','ArrowRight','ArrowUp','ArrowDown'].includes(k);

function addEditable (node) {
  node.classList.add("_arsvia_editable");
  node.contentEditable = true;
  node.focus()
}

function removeEditables () {
  Array.from(document.getElementsByClassName("_arsvia_editable"))
  .forEach(el => {
    el.classList.remove("_arsvia_editable");
    el.removeAttribute('contentEditable');
  });
}

function narrowToSelection () {
  removeEditables()

  const selection = window.getSelection();
  if (!selection.rangeCount) return;
  let narrow = selection.getRangeAt(0).startContainer;
  if (narrow.nodeType === Node.TEXT_NODE) narrow = narrow.parentElement;

  addEditable(narrow);
}

window.addEventListener('keydown', e => {
  if (!isArrow(e.key)) return;

  removeEditables();
  addEditable(body); 
});

window.addEventListener('keyup', e => {
  if (!isArrow(e.key)) return;
  narrowToSelection();
});
</script>
</html>

Other than this worked on the HTML editor some.
Had the machine replace the decorators with actual content that's hidden so that they can be edited.
Still trying to workout how to have it autoupdate the surrounding tag, and deal with insertions, and attributes.
Further not sure how to deal with metadata, scripts, and styling in an elegant way.

>>31313
The reason wrote this was to have an input event fire on the "inner" element.
A far better way to do this is to have the input event look for the commonAncestorContainer of the range.
This is now the entirety of what remains of the lens logic.
The machine found an different way to replace the actual decorator with contents.
That is to have the decorator outside the tree and only position it absolutely when it needs to appear.
There's a small invisible spacer that determines where they are to go in the tree.
Ideally even this spacer wouldn't be in the tree and it's pretty annoying that it's there.
The machine wasn't very helpful other than this, and now seem to barely remember why this helped.
It was something to do with the cursor not appearing in the correct position before if it was 1ch wide.
Got autoupdating of the surrounding tag, and attributes to work also, although attributes are no longer hidden.
A previous version also had the insertion working, but this is no longer the case.
Am starting to be a bit warn from the LLaMa shaving.
Much prefer just programming, and taking things a little slower, but can't seem to.

>>31325
>Ideally even this spacer wouldn't be in the tree and it's pretty annoying that it's there.
Thought of (and wrote) a way around this just positioning absolutely on lines from inner to outer if there's too many tags to display where they belong.
Also managed to get a primitive version of the HTML insertion working (with undo redo), think the behavior that's desired is for '<' to insert '<><\>' with the cursor in the tag.
Am currently working on the display of self-closing tags and maybe head information.
Decided the CSS editor should be a simple button at the end of the tag which opens a popup, ideally rather like the dev tools editor.
That's about it for today.