The neverending quest to rewrite vichan -
Archived threads:
https://archive.is/xiA7y>>26420>the jannies may we involved in a serious effort to write a new chan using C# Blazor soon.Good morning sir.
Can you please do the needful and provide update?
>>26717>typescriptGood choice.
>ReactDoesn't anyone know how to design site layouts in html anymore? If i understand it correctly, React is basically for optimizing infinite-scroll type layouts and batching document tree updates. This might be necessary for webapps with lots of state, but for imageboards it traditionally isn't and they run a lot smoother for it. If the browser caches image and stylesheets, fetching the initial site and updates should be the most common bottleneck, so it's a good idea to keep the document tree as small as possible.
>We will not let people write their own clients>I will document the API in the OpenAPI formatI'm confused on how exactly you want to prevent other people from writing clients.
>Designer's ego is fucking huge and he thinks he does glassmorphism better than anything we shown him.Personally i like a good glassmorphism. Just make sure the site is easily themeable, especially if the document tree is going to be a sprawling, js-generated mess.
>>26759NTA
looks like python and the second one is probably some sort of html/css templating system like jinja. If i had to guess the reason the file is an array is because there can be multiple uploads so multiple files.
>>26757>>26763ChunkedIteratorResult sounds like it is supposed to chunk or "page" the query, can't you do something like
files = db.post_files(thread).all()
Got login working, but broke some other aspects somehow, namely pic related, is breaking deletion… Don't suppose anyone has run into this one before? Am setting via an update and
status = PostStatus[status_string])
but don't see why that should break things this badly. Not too terribly found of Python.
>>26852is this like a convoluted way of implementing enum in python?
can't you just use numbers?
>>26856Tried this out and didn't seem to help any. Am not looking to use them as numbers, just want to be able to test for equality.
>>26856Apparently SQLAlchemy stores the enum as the string of its value so in this case '5' but attempting to use the string of the value in the queries also didn't help any.
>>26858I didn't explain myself. class instances in python are like bags, you can throw anything in them, and sometimes ORMs put state in what would otherwise look like plain data
I looked it up and sqlalchemy apparently does do this. see if you can do "sqlalchemy.inspect(post)" or "sqlalchemy.inspect(post.status)"
>>26859Only some of it's secret because of hardcoded emails, directories, and such. Can show.
>>26860ewww.
>>26860But how do you end up with multiple instances of the enum value?
>>26863What if you try dir(post.status) and dir(PostStatus.DELETE)?
>>26852>>26858>Apparently SQLAlchemy stores the enum as the string of its value so in this case '5' but attempting to use the string of the value in the queries also didn't help any.just dynamic language trash lmao
It's a preference thing but I would never want to write anything in a dynamic loosely typed language, except maybe a lisp. OK maybe forth or some shit like this for funsies.
Good job though, power to you. I don't have time to do anything for myself, let alone a hobby project.
>>26872It has `value', and comparison with that worked. It also has `
eq' and using = calls that.
>>26876tbh I feel like all programming languages and frameworks are kind of clunky at modeling websites. something like an imageboard or a CRUD could be represented as set of layers that do simple transformations on input data. you take the parameters from the request, query the db, adjust the data a little, and then render a template or populate a json payload
imagine how easy it would be to shit out backends if there was something like the unreal engine "blueprints" editor but with something like the C# linq for queries and data transformations. like, a good chunk of *all* code is just boilerplate around the shortcomings of the tools
>>26877>something like the unreal engine "blueprints" editor but with something like the C# linq for queries and data transformations.Was going to compare this to something existing, but there's really no point and they're quite different. It sounds really nice though, and would save a lot of people a lot of work.
>good chunk of *all* code is just boilerplate around the shortcomings of the toolsA shame.
>>26877>imagine how easy it would be to shit out backends if there was something like the unreal engine "blueprints" editor but with something like the C# linq for queries and data transformations. like, a good chunk of *all* code is just boilerplate around the shortcomings of the toolsOnly really works if performance is not an issue, and the backend does very simple stuff imo
BP in UE is a good example, they are amazing to quickly iterate on simple stuff that is not more than function calls. Like they're good to jump if spacebar is pressed, or to fire a raycast and call an interface on the first target if E is pressed, not to implement heavier logic
>>26885a system of roles and actions similar to discord?
a long time ago I used an anarchist ib and one of the boards was made (as a joke) such that all users had moderator privileges. I don't think vichan descendants can do something like that out of the box
>>26887basically all interactions, from posting to banning people are listed as actions or permissions. the "owner" or root user can create roles with arbitrary sets of actions (permissions), and then assign one or more roles to users. so for example there could be the role "poster" with the "post" action, "janitor" with the "delete" action, and "dev" with the "see metadata" and "delete" actions. a single user has one or more roles
note that the action sets can overlap, and that the user can have many overlapping sets of actions
what they did in this case was enable the moderator actions for the default, unregistered user role
>>26897Running into another very stange pythonism bug where logic isn't short-circuiting in a conditional variable declaration. Checks for:
current_user is not None and current_user.is_authenticated
still results in an error of NoneType doesn't have attribute is_authenticated. This was while trying to setup a transition from the login / signup button to a logout button.
Anyway cleaned up the CSS some more so that now the checkboxes seem to have pixel perfect placement, and the post header lines up nicely, with the images properly formatted in all situations. Also made the menu hamburger turn into a big 'X' when clicked (only in CSS).
Hoping to maybe find a solution to the first problem and setup a board management page here soon.
>>26903You can see the subtle change in alignment of the article header and checkbox in comparison to
>>26807. Don't think there are any images of broken image formatting here.
>>26927Christ that took all day! Pic related is an implementation of Vichan style markup using an LL(infinity) parser combinator library called "parsy". My implementation is a worse version of my Java, but it seems to cover all the common cases included cross board links, and urls. The markup syntax needs to change a little, bold doesn't work as is, because it's colliding with italics, and it would be better if the closing and opening tags of spoilers were asymmetrical allowing for nested spoilers. Might be taking feed back on this:
Any changes to the markup you guys are interested in? Before had spoilers as /* and */.
>>26930Decided not to sleep. Added
https://2ch.hk style backlinks, html escaping, made the dark-mode toggle with operating system preference. Realize there is actually a global namespace for post.id, crossing over boards, so might axe the more complex cross-board linking system (despite having just finished putting in the work to get it operational). Changed bold to ##for now##, and spoiler to /*spoiled*/. Would still be interested in what people would change about the markup language.
>>26935>Some sites support BBCode like spoiler and code tags, that's pretty cool.Not too fond of these myself.
Been doing bug fixes mostly, made the images user resizable (no Javascript). Had the idea for a user history page for moderation and personal use, together with an administrator page to control permission, and an already partially implemented page to view reports would be
all of the moderation tooling. Have considered not even having the admin page so that there is no page that's permission restricted. Anyway all of that's on hold because can't seem to get current_user to work all of the sudden.
>>26937>made the images user resizable (no Javascript)This sucked so it got deleted. Feeling pretty exhausted… If had anything better to do it would certainly be dead.
>>26937>Anyway all of that's on hold because can't seem to get current_user to work all of the sudden.This got fixed, and now there is a primitive version of the report and history pages.
>>26945Added a recent.html view which displays threads in chronological order and broke a whole bunch of stuff trying to write the backend for the history page, then subsequently fixed all of this stuff except for the reports.html page which is in a dismal state, don't even know how its supposed to look anymore. Also made it where users are allowed to moderate their own posts (sink, lock, delete, warn). Really reports.html, and admin.html are the only further planned features.
Didn't realize that vichan was
13k SLOC excluding the front-end ($413,000 in developer time according to David A. Wheeler's SLOCCount). Am currently at 710 (nearly $19,000 in developer time), and it seems odd to think there would be something like 18 times the current functionality.
>>26946This is it for the report page.. it's bad.. but am probably just going to leave it as is (if can think of anything better to do.. like resuming the job search). My thought presently is the way it should look is with the reports under the message they concern, only displaying the message concerned once, and these should be bump ordered. An exaggeration but this almost feels like a whole other application. The only thing it might need is to add UserReports in addition to the PostReports currently present.
The way the admin panel appears in my mind is as a two input forms with text areas for emails and a select area for permission to grant or remove depending on which of the two input forms is selected. This lets me continue with the lack of access controls on pages at the expense of the user. This is also not great as the administrator needs to remember the emails of their moderators for promotions and the like.
Lastly another page which might want to be added is a log.html for displaying moderation actions. This would be comparatively easy.
>>26949An even worse admin panel… but with the requested fine grain permission system (discord style), and my own constraint that all pages be public, and secure. Likely the whole project now needs some fall cleaning, (and for someone who's not me to write some Javascript) but it's got all the features that one desired! Kept thinking was going to give up, but guess not.
Current total SLOC count is 820 with a cost estimate of $22,000 once again according to David A. Wheeler's 'SLOCCount'.
Other than pushing to a repo, which just requires me to make an account somewhere (ironic) would anyone like to see any features added, or anything documented?
>>26959Licensed AGPL, and found the right set of hacks to make the API endpoints elegant enough. SWIM also might have lifted some JS for click/highlight to reply, 19 SLOC. Floating reference may also be easy to add, though less so, especially for cross thread references. Entered into discussions about the best place to host a repo
>>26958.
>>26959Added primitive always on autoRefresh (always on every minute) in 20 SLOC and floating reference with another 47 lines of JavaScript (copy-edited from 2ch.hk). Was hoping to add 2ch.hk style endless on-hover replies
as mentioned in https://lainchan.org/%CE%BB/res/26674.html#q29059, but for cross-thread references this requires you to do dynamic lookup of threads in JavaScript, which am not quite willing to do. Next major front-end task for SWIM is the floating quick reply form.
>>26959Added per user granular flood detection to every relevant page except for login/signup and a notice in the title in autoRefresh.js for when a new post is made. None of the features from
>>26959 have been planned, but email verification, and a floating quick reply form are the only even semi-panned features.
>>26981No seriously, it's such a pain to work with. Luckily it's only for four usability improvements: auto refresh, reference on hover, click to reply, and the floating quick reply form.
Have been thinking of how to add these features without Javascript. autorefresh could be done with an iframe and http-equiv="refresh". the floating quick reply form could be just a static form to the left that you can hide or show (sort of like a popup chat menu some sites have). You could make a show-references button to show references without Javascript, though they wouldn't be as nice or granular. And for click to reply one is just out of luck.
Would be interested in other's thoughts on this. The Javascript solution for all of these things is probably the easier of the two.
>>26982just use javascript. iframes are incredibly dated. The fact is you should just use javascript, if you need a library try some modernized jquery alternative like alpine.js or htmx
the only alternative to this is just biting the bullet going full SPA and writing a frontend in angular/react or something, or thin client shit like C# blazor. Webasm is a meme, iframes are from the 90s. you are forced to use javascript unless as said before you go full SPA and transcompile typescript, or something even edgier like clojurescript, rust to wasm, or even more obscure shit like dart.
>>26982>the floating quick reply form could be just a static form to the left that you can hide or show (sort of like a popup chat menu some sites have).this is how nanochan/picochan does it.
>>26984what part of iframes do you specifically consider dated? the fact they have well defined privileges as a potentially insecure third-party resource?
>>26984Even if it was reduced there would probably still be some on my board since there are some usability enhancements which can't be had any other way. There doesn't seem to be a need for any libraries however, still don't actually know what JQuery does…
>>26985>this is how nanochan/picochan does it.Floating quick reply went really poorly, and not much would be lost scrapping the present attempt and going the other way.
>iframesThe only disadvantage of this for me is that it pollutes my views.py because every page with autorefresh (and thusly an iframe) would need a separate route.
>>26986>iframesAnother disadvantage is that without Javascript can't think of any way to change the title when there are new replies
or rather to change it back to the default state once they've been viewed so the autorefresh would be of limited utility.
>>26993>Might give it a go.Am just going to reload the whole body
well <section> of the thread every time there's an update for now. Have everything setup except for some reason it seems that each socket handler only executes one at a time.
>>26994>>26995>allow some of these options to be passed in the url.This seems possible, what other features would you like to see disabled?
>>26996theme selection, (you)s and inline image/video expansion seem like good options.
on that note: with vichan every theme, except the default and yotsuba b, has a noticeable delay for every page, so you might want to verify if themes get cached and if there is significant latency during loading. i briefly copied the css of a theme into a user stylesheet, but it broke often because of site updates.
>>26996>Have everything setup except for some reason it seems that each socket handler only executes one at a time.Really confused about what could be going wrong here.
>>26997>theme selection, (you)s and inline image/video expansion seem like good options. This sounds like a great idea, apparently this is a JavaScript feature which surprises. Themes and inline image expansion aren't implemented yet, and (you)s are in the backend, but can probably make all of this work.
>with vichan every theme, except the default and yotsuba b, has a noticeable delay for every page, so you might want to verify if themes get cached and if there is significant latency during loading.This would probably be the job of whatever was in front of the application. Themes are very small currently though, just a change of around ten CSS variables.
>>27004Got Anonymous posting, address based logging, flood control, and bans implemented. Would like to have it where it was a per-board configuration option for Anonymous posting, but don't see how it could be easily implemented at the moment.
>with vichan every theme, except the default and yotsuba b, has a noticeable delay for every page, so you might want to verify if themes get cached and if there is significant latency during loading.Have been trying to solve a similar issue to this where there are a few milliseconds of unstyled content which flash when using the theme selector. Apparently the only way to fix this is to modify things server side to work with the search parameter.
>>27007>Would like to have it where it was a per-board configuration option for Anonymous posting, but don't see how it could be easily implemented at the moment.This is now implemented. Source including the frontend is now at around two thousand line. The whole thing probably needs a once over, and a lot of usage to find any missing bugs and inconveniences. Here's a list of my todos at the moment:
Clerical
- host source
- public instance
- project name (presently "cyberia")
Source
- word filters
- captcha (after some number of failed login attempts, and on signup)
- email verification or validation
- reimplement themes in backend
- views for user and address reports
Am currently implementing the captcha system. Might finish it tonight. Think could get away without email verification if this was done right. There's currently no plans for range bans, that might be a mistake, wonder if they're used often in practice.
>>27025Yesterday gave the whole program a read through correcting errors along the way. Also expanded on the todos a little. Managed this morning to get thumbnail resizing setup to save on bandwidth. Would like to setup incremental autorefresh next, but don't exactly know how to do it without dropping messages, especially considering there are bulk operations.
Clerical
- host source
- public instance
Backend
- views for ban reports.
- easier way to create the admin account.
- LLM for moderation. (Llama Guard)
- prettier error pages.
- incremental autorefresh
>>27030Styled the error pages, cleaned up the error handling in source, and add proper handling for 500 errors in the mean time. The incremental refresh is turning out to be quite tricky. Was thinking could take a hash of the page to check for dropped messages, and just refresh the whole thing
like is currently always done in this case, but it turns out the preview script modifies the DOM and this is no longer a reliable measure. Am open to suggestions on how
of if its necessary to do this. Is the gevent loop just fast enough? The LLM has to wait for me to get a new computer to avoid swapping.
>>27030The only addition to the program after this morning was the simpler system to create admin accounts. Realized to setup incremental autorefresh would require reparsing posts to generate backlinks and at that point gave up for now. The view for reports of posts, users, and addresses requires me to make a query that don't know how to express in SQLAlchemy
think it would be an outer join, and coalesce. The LLM requires me to have a better computer to avoid swapping while developing. So that's it from me for now. Releasing the source is the only thing which seems possible for now, and still haven't decided on a platform for this.
Open to any suggestions.
>>27018I suggest you add the ability to run messages through an arbitrary server-side spam detection program. See
https://bogofilter.sourceforge.io for something like spamassassin, that also works for plaintext.
>>27045>ability to run messages through an arbitrary server-side spam detection program.Added, though this is a feature for advanced users given the implementation. Will use this for my LLM moderation system and have it as a separate program.
>>27045>beg for money.In the lupen sense and not the grifter, went ahead and removed it from the temporary nav.
Not a clue what boards to actually include either if it ever gets hosted.Also added stickies and locked stickies, styled the success pages, and cleaned up the error reporting some more.
>>27062Thank you! Avoiding the LLM is probably more practical because of the 2Gb of ram it eats up raising costs 50%. The idea was just to make moderation a little easier for those of us without thick skin.
>>27059Haven't yet found a way to write an ActivityPub server that would be reasonable. The FChannel implementation is many thousands of lines. The way the've implemented actors is also somewhat strange. Rather than having actors be a salt hashed address or a user account they've made them boards:
https://github.com/FChannel0/FChannel-Server/issues/9#issuecomment-822675142 There is a good JSON-LD library from the looks of it.
>>27134Don't suppose anyone here knows how to make SQLAlchemy return only the first image for each thread? So far as can tell am doing everything correctly here, but still getting an error that "row_number" doesn't exist. If it were just SQL could pull it off but for some reason the "label" even as it emits an AS does not cooperate with the following use of the labeled item.
Unrelated am also having an issue with Starlette-Login trying to serialize user to JSON for some reason and failing at this. Should be able to take a deeper look into this myself at some point.
>>27168Is there someway to have two hashes for the same object and then confirm that they are for the same object without knowing what that object is? Was thinking for moderation purposes it would be interesting to have hashes of addresses communicated between federated servers along with their respective messages in such a way that each could check whether or not its an already banned user. Found this:
https://crypto.stackexchange.com/a/102963 but it seems a little theoretical, so probably not.
>>27170>Hopefully sometime tomorrow can finish off the migrations with its comprehensive testing,Wouldn't say the testing was quite comprehensive, but did a quick read through, and some sporadic testing which turned out well. Went ahead and merged the Starlette branch with main.
>>27170>begin to implement new features again.Am considering getting rid of the (you)s to enable caching to work more effectively. Theoretically post identifiers could be stored in the session, and this way the (you)s added back at a later date. Am also considering replacing the server side rendering of the [logout] button with client side rendering. The downside is slightly less functionality for noscript users, the advantage is that caching is much easier.
>>27174Made the image and reply count appear on the catalog page. Also added the user and address reports to the reported page. Been working through bugs otherwise. Here are my remaining TODOs:
Devops- host source
- public instance
Missing- RSS feeds.
- ActivityPub.
- post search?
Most of what the program needs at the moment is testing, to uncover bugs and then subsequently fix them. Have been doing some of this already today, but it may be about time to finally get the source code hosted somewhere and start receiving feedback. Biggest hold up here is deciding on the platform; seems like it's GitHub or Codeberg.
>>27178Don't have it in me to write documentation at the moment but here's the repository:
https://codeberg.org/jung/arsvia Plan to add install instructions, and some pictures. The gist is:
1. install postgres, imagemagick (libmagickwand-dev), python3, and pip.
2. run pip install -e .
3. setup a user and database in postgress and add to config.py
4. python3 arsvia
5. navigate to 127.0.0.1:5000
6. create an account which will automatically become an admin.
>>28343Make everyone register an account
(for real tho this could be not bad. It would help against spam and illegal content by raising the barrier to entry some [and new account creation could be temporarily halted, which is better than harming vpn or tor users or etc], and on the front everyone could still be anonymous. Or a tripfag at their choosing. whatever. Would it be so bad?)
>>27187Am considering rewriting arsvia, my imageboard, in typescript (using Next.js and TypeORM) to expand my techstack.
Arsvia has user pages containing all their posts, and thought to run with that idea a little.
Namely adding a feature for sharing usernames either with specific people or making them public.
And more interestingly, to allow users to assign arbitrary nicknames to other users, and to post to user pages without a board.
Guess the user pages could reuse the moderation functionality so that users could manage their own pages.
Maybe with the Oath2 JWT authentication and try to figure out email verification too.
In short it would be something in between an imageboard and a microblogging platform.
>>30504Am thinking for this version instead of using the UUIDs directly I'm going to rewrite in the render such that it generates a sequential ordering by creation time followed by internal UUID. For threads there could then be an array perhaps with ranges or an epoch to avoid the ordering reusing numbers for the render (pretty ugly imo).
Further I'm going to use threads as namespaces like in textboard.org so that each thread starts from zero.
I'm also considering using the UUID primary key of the address log table to emulate accounts for anonymous users. This would mean that they could be federated if that was something of interest.
Could consider also using readable UUIDs like
https://github.com/Debdut/uuid-readable though this is a little chaotic, but at least they wouldn't be like Ganymedean's from PKD's "The World Jones Made".
>>30505Interesting. While working on the successor to
>>>/siberia/679700 i settled on using a tai64n timestamp with 2 extra bytes of entropy (prepended to the start, so posts and threads can be distributed into buckets). This also means my thread files, which consist of lines of these IDs, have fixed-size records and can be atomatically added to using O_APPEND.
>>30510Downloaded, and read the source for the three main SSG programs, and skimmed io.c.
It's obviously a very interesting approach.
Just from reading what you wrote here for some reason thought had made the program to use fixed disk space.
That it's a textboard has reminded me that I've no idea how to federate files.
>>30514Thank you. If i had to summarize this approach, it would be reducing complexity by finding the right interfaces.
>That it's a textboard has reminded me that I've no idea how to federate files.Elaborate.
>>30515>right interfacesWouldn't have guessed that this is what you would have called this.
Don't worry about the files thing was just being foolish.
Should probably spend some time learning UNIX a little better.
If for no other reason than to get better at using other programming languages' standard libraries.
>>30514In other news seem to have gotten something like a nearly complete schema setup:
auth
├── PrivateAddress.ts
├── PrivateUser.ts
├── PublicAddress.ts
├── PublicUser.ts
└── UserRole.ts
reports
├── PostAction.ts
├── PostReport.ts
├── Report.ts
├── UserAction.ts
└── UserReport.ts
threads
├── File.ts
├── Nicknames.ts
├── PageInfo.ts
├── Post.ts
├── Reference.ts
├── Tags.ts
└── UserProfile.ts
3 directories, 17 files
The most interesting features are the ability to assign nicknames to anonymized (by default) IP addresses or users.
And the replacement of boards with moderated user created tags; along with the posting to multiple tags.
Am thinking of federating anonymous posts using throwaway
https://github.com/Debdut/uuid-readable UUID usernames.
This would be as an alternative to the high dox potential of having anonymous histories based on IP.
>>30516>to get better at using other programming languages' standard libraries.C doesn't really have the best standard library. You usually want to avoid using anything that isn't a thin wrapper around a syscall.
>Wouldn't have guessed that this is what you would have called this.I got this from
https://skarnet.org/software/skalibs/djblegacy.html<One of the "DJB philosophy" key points is to question the interfaces. You have a task to do; you have existing interfaces. What do you do?<Interfaces should be questioned right down to the libc. You cannot build strong software on flakey foundations. And from a system and network programmer's point of view, one thing is clear: most standard libc interfaces suck. There is no buffered asynchronous I/O. There is no timed I/O. There is no heap management helper. Even simple system calls are not always guaranteed to succeed!UNIX doesn't have proper records, which most developers compensate for with databases, essentially a second filesystem layer. Files and directories are really the only level where database operations are simple, reliable and fast, so if you keep your state simple enough, you can map all of it onto simple file creation and access.
The next version will allow composing posts by processing arbitrary blobs, but i'm proud of how the static site generation approach worked out for this one. The locks are the only non-essential state of the system and usually short-lived, everything else is either persistent data, static configuration or entirely ephemeral.
>>30516Am probably going to restart.
I've realized this should be a ActivityPub server first.
And only secondly a anonymous imageboard (front-end).
>>30517Wonder if a torrent client might be candidate for a good djb application.
You could parse the bencode into the filesystem operating as a key value.
Lists would then just be dictionaries with enumerated keys.
Unpacking the data is probably more elegant in C than in other languages.
Clients are largely well segmented with clear modules based on a request/response pattern.
These could easily be turned into program boundaries, each implementing the previous response struct/unpacking.
>>30602My gripes with other clients have made me look at the spec before, but it's a lot and i'd rather finish the ftp server i have lying around somewhere first.
The idea seems good and would work well with a lot of simple multiprocessing patterns. Personally i think the connectionless nature of the protocol would lead to a lot of process creation overhead with a forking model, so i would instead have a fixed number of worker processes servicing these. Maybe these workers could also spawn a process for each torrent and forward requests to them, which could then be treated as a heartbeat to approximate active "connections", if that makes sense.
>>30696You're correct.
>>30697This is what was thinking:
There's a daemon to check the tracker, establishing connections via pipeline, a program per request each subsequent program unpacking the response.
Ultimately spawning a message handler per torrent client for some subset of the clients in the tracker list.
A list of active clients should be persisted, along with a table of pieces as a lock, and information concerning global resource utilization.
Handlers commit process seppuku, or choke, if they are too slow, judging by global resource utilization, it's similar with upload.
These allow the "central" daemon to be exited and spawned periodically for example in a cronjob if that's desirable.
>>30699I didn't think about the fact ip addresses can basically act as session identifiers for clients, but that negates some advantages of UDP. While socket-level broadcasting isn't implemented on linux, i don't feel like this way to track sessions makes for good program architecture.
There are two cases where performance matters, which are many clients leeching few files and many clients leeching many files. The forking model is bad for both, while something to optimize specifically for the first would handling requests in a tight event loop. It shouldn't even matter if the server spawns a process connecting to it for each request or directly exchanges requests with it.
>These allow the "central" daemon to be exited and spawned periodically for example in a cronjob if that's desirable.This isn't as important as it might seem. Polling periodically is inefficient, but if you have a fixed number of processes listening on a specific event on a file for example, you're not going to consume any computing resources or a lot of memory while they aren't running (UNIX is a timesharing system after all). This wouldn't be a bad choice as a way to ensure your daemon is reentrant though.
>>30804>>30805>>30821Ended up taking a break from this project, and school and work are about to kick in so it's probably for the best.
The reasons for picking Typescript are as follows:
1.
https://fedify.dev/ is way better than the python alternatives.
2. Wanted to be able to apply to more jobs, was thinking it was a question of quantity (probably wrong).
(The fresh Island architecture and preact also seems nice.)
Seems like my contact fell through so am back to programming. arsvia-redux has expanded and transformed dramatically in scope - as vaporware. The idea now being a social knowledge base with some AI integration, and anonymous content. Meanwhile an arsvia rewrite seems underway using some of the lessons from
>>30995 but hilariously in a bottom up, test later manner. Might actually make a hosted instance for dogfooding purposes this time. Truthfully I'm much more interested in the much more difficult redux project, but this is still good practice, and it's almost outlandish.
>>31199Have already run into issues with what really should work as far as SQLAlchemy validators go. The following seems impossible:
@validates("thread")
async def _validate_thread_in_db (self, thread: int) -> None:
thread_post = await self._session.execute(
select(Post)
.where(Post.id == thread)
.where(Post.thread == thread)
.where(Post.status != PostStatus.DELETE)
)
if thread_post is None:
raise ValueError("thread no longer exists, or never did.")The issue is that it can't validate async functions. Might try again tomorrow or soon with SQLModel which supposedly can manage this sort of thing.
>>31200>SQLModel which supposedly can manage this sort of thingThe closest solution found was here [^1], but it incompatible with SQLModel.
Am considering using Pydantic and aiosql [^2] instead,
just for this.
Sort of postmodern in a good way, and imagine it would require fewer hacks.
That is if have the energy to do non-trivial refactoring in this rewrite.
:[^1]
https://github.com/team23/pydantic-async-validation:[^2]
https://nackjicholson.github.io/aiosql/ >>31203This is very kind of you James.
There wouldn't be much of a reason to run this version two arsiva.
Your styles wouldn't apply, and there's no migration scripts yet.
Even if it all worked you just wouldn't be getting much beyond vichan.
But arsvia-redux could be interesting if you wanted to restart and I had the energy.
Main problem is it seems like it would just be a ton of work, and it's still in planning.
>>31200>>31202Think the basic problem with this validator is that its
relational, but in the class.
It's the same problem with the handling of bumping and locking of full threads.
There's still no way to slice through the circularity of File and Post validation.
Closest seems to be to have a single insert and validate function.
Hints at a larger PostWithFiles class.
>>31208The following seems to work well.
Unfortunate that resolve_references is still in the init function.
This makes the functional core a little smaller.
class Post (BaseModel):
metadata: PostMetaData
references: List[Reference] = []
files: List[File] = []
comment: str = ""
@model_validator(mode="after")
def _validate_files (self) -> Self:
if self.metadata.thread == self.metadata.id and len(self.files) == 0:
raise ValueError("zero files provided, expects at least one.")
if len(self.files) > config.BOARDS[self.metadata.board]['MAXIMUM_FILE_COUNT']:
raise ValueError("post exceeds the maximum file count.")
return self
def __init__ (
self,
connection,
board: str,
thread: Optional[int],
user: Optional[UUID],
address: UUID,
subject: Optional[str],
file_storages: List[FileStorage],
spoiler: bool
):
# XXX: Until insert metadata lacks the thread, and id.
self.metadata = PostMetaData(board, thread, user, address, subject)
for file_storage in file_storages:
self.files.append(File(self.metadata, file_storage, spoiler))
parse_tree = parse_with_partial_references(self.metadata.board, self.comment)
# XXX: We must query for the thread of posts referenced to form link.
parse_tree, self.references = resolve_references(parse_tree, connection)
self.comment = render(parse_tree) >>31212Moving in the direction of witness objects containing the bulk of the validation, and lots of frozen classes with static methods, keeping the imperative as small as possible, and separated though not in separate modules. Key difference from the plan is starting (probably erroneously) with the types, an validation logic.
That is start by structuring the state-space. There's also some "separation of concerns" between validation of the object, the object, and the persistence layer, whereas was planning on just having a massive single object for the domain model.
>>31221Had to unfreeze all my dataclasses which is a bit annoying. Renamed the Object classes to be ObjectMixin which is a little more appropriate. Also was able to rewrite the schema today, the results in a viewer rendered attached. It's probably not going to be too difficult to get the queries to work too but building the Repositories may take a second. A sample of what (the simplest of) these look like follows:
def get_thread_post_responses (database: Database, board: str, thread: int) -> List[PostResponse]:
posts, files, references = await asyncio.gather(
PostRepository(database).get_replies_by_thread(board, thread),
FileRepository(database).get_files_by_thread(board, thread),
ReferenceRepository(database).get_back_references(board, thread)
)
post_files = {}
for file in files:
if post_files[file.post] is None:
post_files[file.post] = [file]
else:
post_files[file.post] += [file]
post_back_references = {}
for reference in references:
if post_back_references[references.referee] is None:
post_back_references[reference.referee] = [reference]
else:
post_back_references[reference.referee] += [reference]
post_responses = []
for post in posts:
post_response = PostResponse(
**post,
back_references=post_references[post.uuid],
files=post_files[post.uuid]
)
post_responses.append(post_response)
return post_responsesSad.
>>31240This is a good function, think it's going to do the bulk of the lifting for my unmarshaling to larger objects.
from functools import reduce
from typing import Any, Dict, List, TypeVar
T = TypeVar('T')
def group_by_field (xs: List[T], field_name: str) -> Dict[Any, List[T]]:
def f (acc, x):
k = getattr(x, field_name)
return acc | {k: acc.get(k, []) + [x]}
return reduce(f, xs, {}) >>31251Dictionary union is apparently dog slow…
https://perfpy.com/1058I've switched to your version anyway, as it's also easier to understand.
>>31253This doesn't even seem like it should be true! It's not that it's binding to the empty list conditionally that's slowing it down (it might even be a hair faster to do this
https://perfpy.com/1060). Not sure what this thing could be doing, and why seemingly pretty basic abstractions would be slowing the thing down.
>>31256result.setdefault(key, []).append(x)
Doing this in the hot loop means allocating an empty list in every iteration, even if `key' is already in result. It's probably why it is slower.
In case of #1060, there's an error for Code 2 and rerunning the whole thing a few times, it seems random which comes out as the winner. The memory is randomly changing between around 700 KB and 1.06 MB, something is slightly off.
>>31267Had an unscheduled break from this project today, hopefully it doesn't last.
Or otherwise get some momentum to switch to arsvia-redux (even better).
Would like 2ch.hk style recursive post preview.
This requires an additional endpoint to fetch individual posts.
It would also require JavaScript, which is making me reconsider.
Would also like to have the live-board feature be incremental to save on bandwidth.
Would almost be easier just to rewrite it in Fresh…
>>31272A dynamic static site generator!
>>31274Had a couple funny ideas: one about using HTTP Digest Authentication with passwords stored in /etc/shadow and an endpoint for registration which generated a OTP and sent it over email. Another was to use xattr for properties like file or reply count which wouldn't be as elegant in the file name.
In a different context was that it might be useful to store records like this for my redux application since archivability was one of the objectives. Just not sure about the format which would be desired for users to archive replies to their posts, or if that should be ignored entirely.
Found
https://writefreely.org which is similar in many ways to my concept for redux. It seems like a good project and I'm glad it exists, but it's missing some of my features:
- OP moderated "threads"
- federated replies
- file versioning link-rot protection
- wiki-style links with user directories [[@user@instance/filename/version]]
- back-links
- distinction between zettlekasten and bibliographic notes
- AI "integrate knowledge" button.
- anonymous posting
This is a ton to implement so don't feel to anxious sharing on an esoteric Mongolian picture site.
Would love to discuss here any additional features you'd like to see in such a application.>>31282Ugh, somehow forgot one:
- archivability of all content, and someway to archive replies to your content.
>>31282I haven't looked into your projects, because i don't have a grasp on any of the frameworks
or even the language. What it sounds like to me though, is that you're trying to write a kitchen sink chat/bbs/wiki/booru from the top down, which looks like a bad idea to me. First, if someone needs only one of these 4 main functions, they will rarely consider using your software over a more minimal and consequently simpler alternative. Second, forum software usually doesn't sit in a vaccuum. It exists to serve the need of a community and unusual functionality often accrues over time as the need arises.
When i write software, i try to minimize the set of core features while supporting as wide of a featureset as possible. This is the case for most programming language runtimes, which are consequently relatively easy to port
even c/unix given the scope and also emacs. Your system may be robust, comprehensive and elegant, but to me it seems like an evolutionary dead end.
>>31282>>31283so activitypub (+ some trick to obfuscate user identity) but with mentally retarded non-features: ai and hardcoded formatting
if it is federated and I want to setup an instance I have to also setup a local chatbot or pay a chatgpt commercial license? that's an absurd entry cost for what will be, at best, an extremely niche platform.
"link-rot protection" is always a meme because if I stop hosting something, it doesn't matter what the url was. if by link-rot you mean that content that is still hosted by the server becomes unreachable because of url changes, every single ib that stores files by their md5 hash (like 4chan) already has that feature. matter of fact, this is only a problem in enshittified corporate internet. do you also need to specify that your ib won't be vulnerable to script injection?
>>31284>kitchen sink chat/bbs/wiki/booruI think there is a scope, but it might be too big. Maybe the scope is rather a purpose, it is what it does. Anything which would improve my usage of my (and others') zettelkasten. [^1] (To put briefly something put much better by me in a different format today) I'd like to improve thinking.
>i try to minimize the set of core features while supporting as wide of a featureset as possibleGuess the challenge is in designing this… Am struggling to imagine what to get rid of!
>>31286>hardcoded formattingThis is a very interesting critique to me. I don't think it would federate well if you had any options for formatting. You could certainly allow for customizable CSS and scripts though, just like a static site generator.
>if it is federated and I want to setup an instance I have to also setup a local chatbot or pay a chatgpt commercial license?No, this was going to be an optional feature, disabled by default. :[^1] If there was some tool to help me finish books I've started taking notes on that would be even better, but this and the social impetus (of publishing) to do the right things, are the only things to mind that could encourage me to use a zettlekasten better.
>if I stop hosting something, it doesn't matter what the url was.Haven't worked out the concept yet but it would be nice to have local archives of linked markdown pages (if this is recursive it could cause trouble). I believe foreign replies would be locally archived by default since they're sent to the inbox. And another idea was to send a reply event every time there's a markdown link for backlinks (but this would be impossible to do recursively).
>>31288So if you make an Article instead of a Note you get unlimited length and HTML tags because you just link back to your instance in the feeds. Am reminded of my hatred for traditional WYSIWYG, and the following from six years ago is accessed in memory (pic rel):
https://secretgeek.github.io/html_wysiwyg/html.html Bet could have an HTML editor that, like Obsidian LaTeX rendering, is rendered typically when the cursor is outside of them, and otherwise in this "brutalist" view. How hard could it be?
>>31291Was playing around some more with this idea.
- Walk the tree
1. getting the tag name and attrs, setting :before and :after as needed to a generated class name.
2. setting tag decorator position: absolute and opacity: 0 by default
3. applying contenteditable=true to everything.
- When the cursor is inside or adjacent a tag display it. (only implemented currently on :hover)
- On insert create a new Node if that's necessary
- Some interface for editing existing tags???
This sort of thing gets a little hairy for more complicated websites.
Might be nice if the decorator was overlaid.
You could then have a toggle with perhaps a mode for all, or just the inside/adjacent.
>>31296Before i tried wrangling HTML into an embeddable markup format, i would just bite the bullet and define my own markup dialect and write transpilers from a HTML subset, bbcode, etc.
Look, if you can't think of an elegant base framework for your features, i'll try doing the work for you:
An article contains a body (content), terminal tags (title, date, flag, uploaded files) and non-terminal tags (forum-style tags, thread number, thread number, tripcode). Non-terminal tags define relations between articles, allowing a client to submit posts to a thread on a given board or display all threads in bump order. Tripcodes act as a secure identifier and therefore allow user accounts or pages by assigning articles to an author or even submit a twitter-style thread under a disposable trip.
I think the above definition is concise enough to be a small subset of something like activitypub (the only "activities" would be submitting articles and retrieving articles), but general enough to implement everything you mentioned. User sessions, link walking and display would all be malleable, i feel like encoding all of this into a stripped-down format goes a long way though. Now it's your turn to build the next layer!
>>31298To give you an idea about what this would look in practice, i will now define a few of your features on a very limited set of non-terminal tags: board, thread and tripcode:
The server is free to reject article submissions to boards or thread that have no previous representation or "don't exist" in whichever representation the server chose. The absence of a thread tag assigns the article a free thread id. Tripcodes act as user identifiers, therefore there can be a convention of showing the latest thread under the trip on /~users/ as a user page (allowing only one thread id per trip on this board and overwriting it on new submissions would make this more convenient). 'Recpients', a terminal field containing a list of tripcodes, would allow the server to discriminate who could retrieve articles, which would be necessary for implementing private messages (i would also dump these to a special purpose board).
>OP moderated "threads"This one is tricky, but the most straightforward approach would be submitting to a moderation queue board and having the OP "resubmit" accepted articles.
>distinction between zettlekasten and bibliographic notesSpecify a terminal tag for each.
This scheme is fairly kludgy, but i think it gets the point across.
>>31298The idea that's floating in my head (now) is to not embed HTML, but the application, as a set of optional widgets.
One widget displays backlinks as Article links, and replies as Notes, and allows for the creation of Notes (maybe three tags for these).
Another takes an inbox, outbox, or sharedInbox and creates a feed.
The base front-end for the application is a nearly full-screen modified contentEditable with a submit/edit button.
Authentication might be a little tricky… but should be possible…
I do think the distinction between relational data and formatting data is valuable.
Even though this is a smaller application, it still feels like it could be smaller.
>>31298The parsing analogy is interesting. Think it's understood. Wonder if it generalizes?
>>31299So this is about API design right?
You're saying there should be, in this case, a single endpoint.
With hidden fields in the form and the action attribute set.
The point is then to reduce the complexity of the transmitted data, endpoints, and render.
All while achieving a certain level of civilization.
This makes me want to get back to rewriting arsvia2, instead of idly contemplating redux.
Today only managed to get nested contentEdtable elements to work like they should.
The key idea is to zoom out the lens (all the way) for navigation and zoom in (all the way) for editing.
Focusing the body with addEditable is necessary for repeat keypresses.
<!DOCTYPE html>
<html>
<style>
*:focus { outline: none; }
</style>
<body>
<div id="outer">
some text
<div id="inner">
inner
</div>
more text
</div>
</body>
<script>
const body = document.body;
const outer = document.getElementById('outer');
const inner = document.getElementById('inner');
inner.addEventListener('mouseup', () => narrowToSelection(), true);
inner.addEventListener('touchend', () => narrowToSelection(), true);
outer.addEventListener('mouseup', () => narrowToSelection(), true);
outer.addEventListener('touchend', () => narrowToSelection(), true);
const isArrow = k => ['ArrowLeft','ArrowRight','ArrowUp','ArrowDown'].includes(k);
function addEditable (node) {
node.classList.add("_arsvia_editable");
node.contentEditable = true;
node.focus()
}
function removeEditables () {
Array.from(document.getElementsByClassName("_arsvia_editable"))
.forEach(el => {
el.classList.remove("_arsvia_editable");
el.removeAttribute('contentEditable');
});
}
function narrowToSelection () {
removeEditables()
const selection = window.getSelection();
if (!selection.rangeCount) return;
let narrow = selection.getRangeAt(0).startContainer;
if (narrow.nodeType === Node.TEXT_NODE) narrow = narrow.parentElement;
addEditable(narrow);
}
window.addEventListener('keydown', e => {
if (!isArrow(e.key)) return;
removeEditables();
addEditable(body);
});
window.addEventListener('keyup', e => {
if (!isArrow(e.key)) return;
narrowToSelection();
});
</script>
</html>Other than this worked on the HTML editor some.
Had the machine replace the decorators with actual content that's hidden so that they can be edited.
Still trying to workout how to have it autoupdate the surrounding tag, and deal with insertions, and attributes.
Further not sure how to deal with metadata, scripts, and styling in an elegant way.
Unique IPs: 38