[ home / rules / faq / search ] [ overboard / sfw / alt ] [ leftypol / edu / labor / siberia / lgbt / latam / hobby / tech / games / anime / music / draw / AKM / ufo / 420 ] [ meta ] [ wiki / shop / tv / tiktok / twitter / patreon ] [ GET / ref / marx / booru ]

/tech/ - Technology

"Technology reveals the active relation of man to nature" - Karl Marx
Name
Options
Subject
Comment
Flag
File
Embed
Password(For file deletion.)

Check out our new store at shop.leftypol.org!


File: 1726459786963.png (365.18 KB, 709x538, nuimageboard.png)

 

The neverending quest to rewrite vichan -

Archived threads:
https://archive.is/xiA7y
270 posts and 59 image replies omitted.

>>32256
>I kept getting bogged down before in my dislike of in effect having two different account systems: one username, and password pair for moderation (session based - in part for protected views), and a (optional) trip, and password pair (form based) for posting.
I guess the merger we've just been discussion is roughly that there could be session based accounts for Actors (how you get a trip) and deletion passwords for posts, with the password hash. You can delete or edit either by the post password hash or by the session login.

>>32256
>you could use name for the display name with preferredUsername set to the secure trip which is "unique"
In the general case there are normalfags on the site, which wouldn't want to link to user pages by a tripcode.
>An advantage of the tags over board approach is that it makes clear which Group is posting, the only one that exists locally, if you want to reach out into the rest of the fediverse for posting.
In my planned approach boards are immutable, supersedable anchors, that allow threads to link to them. Therefore a different instance could create a post linking to a board or a thread (anything allowing replies) and every node in sync will render it on the board in accordance with the implicit graph structure.

In this case unstructured, tagged posts should probably only be propagated on small nodes or use a local tag, which may be shown as a "timeline".

>>32258
>for normalcomards
Well, you're right this isn't ideal. Mentions would be uglier even if you had autocomplete from follows/thread actors and info onmouseover.

>immutable, supersedable anchors, that allow threads to link to them

Excellent. But is it simpler, and more interoperable than just having a custom of mutual following for Group actors?

>>32259
>custom of mutual following
I don't think follows should be part of the protocol. My scheme would use replication structure i.e. readers are either downstream nodes or exposed out-of-band by another node. Prose is highly redundant, so i think native lz4 compression would prevent even a several decade-old network run on top of the protocol from reaching the same multi-terabyte data sizes as usenet. If storage size was to blow up though, node admins should still be able to pivot to a more frugal replication policy, like most activitypub instances have by design.

>>32261
>I don't think follows should be part of the protocol. My scheme would use replication structure i.e. readers are either downstream nodes or exposed out-of-band by another node.
I don't fully understand, and admittedly my abilities to implement complex programs are limited, if this would in fact be one.

>native lz4 compression

This is an important point. I believe you can do this for postgres rows.

I do now have a program (technically called "arsvia-redux") which has the creation of Person objects for login and signup using HTMX, sessions, and CSRFTokens. It needs testing, review, and revision. I'm not sure I really have the energy to dedicate to it or not.

>>32268
It seems like eventually the AI will be better even at Marxist analysis than experts. I'm very interested in making a censor. It's a bit of a stretch goal with this project, even if Llama Guard is central. I don't feel exactly qualified to make it. Censoring chauvanist content and policies seems obvious since it's so prevalent. The truth is that there are so many errors possible, and some of them are so subtle. It seems almost impossible to censor them all.

>>32279
>AI will be better at Marxist analysis than experts
It can be useful at flagging things for human review, but please don't drink the AGI koolaid. LLMs are fundamentally incapable of reliably performing tasks outside their "training" data.
>>32268
>I don't fully understand, and admittedly my abilities to implement complex programs are limited, if this would in fact be one.
To get around the limitation of immutable records, child records ("scions") will reference a field of an existing record ("stock") and on sync get added to a mutable substructure ("shadow") of the stock. This means requesting scions will only turn up records present on a particular node, but it also means networks have an interest in propagating as much metadata and inline data as possible to every node.

>>32280
>It can be useful at flagging things for human review, but please don't drink the AGI koolaid. LLMs are fundamentally incapable of reliably performing tasks outside their "training" data.
Llama Guard 4 still has 11% false positive rate, but I think eventually this would be low enough to serve as a filter for content without moderator intervention, this is actually critical to the design because federation makes CIDR bans impractical.

Perhaps I'm getting habituated to the trolls here, but after the spam, and since the plan is to be text only, part of me is even more interested in censoring wrong thinking than illegal content. Yet I've no idea how to formulate such a censor.

The point wouldn't be to create a hugbox, where there was strict decorum, and you couldn't call each other uyghur or whatever. There's still plenty of disagreement among the left, and analysis to current events to talk about. So what is the point? I guess the idea is just to at once allow for outreach to a larger community without being overwhelmed by wrong thinking.

This is related to the follow-follow model I proposed. If the Group follows any Actor that follows the Group, and you allow users to act as the Group with other Actors (perhaps there's an account to make a user feed, but they can still post as the Group - and these posts then show up in the Group feed) you can get outreach without having the main page filled with wrong thinking.

>>32282
>Yet I've no idea how to formulate such a censor.
My best guess is that the best way to formulate this censor is not by ahead of time trying to think of an exhaustive set of rules but by labeling the dataset and having the machine derive the prompt ala: https://dspy.ai/#2-optimizers-tune-the-prompts-and-weights-of-your-ai-modules This would be a lot of work.

I've made a repository for this federated rewrite, in case anyone is interested: https://codeberg.org/jugaad/arsvia-redux so far it has the basic machinery for the creation of Person objects, session management (>>32268), and the improved (>>31540) parser, with a lot of tests.

I'm thinking presently that the moderation would be a challenge. I really hate thinking about this stuff let along actually managing a site. With PhotoDNA, URL rules, and word filters might be possible to remove the most offensive content. Llama Guard can catch some other junk, but there's no human out of the loop solution that exists today, and that there are no accounts, and federation (without federated CIDR lists) makes any sort of bans effectively impossible.

>>32438
Anyone have any ideas what to do about bans?

>>32439
Have been working to try to get the Note/Article/Page TABLE setup, and the subsequent catalog view. Am just using picoCSS at the moment but pics rel. are what things look like right now.

>>32439
When all else fails, make users keep a killfile! Vichan can hide by name, which is obviously of limited utility when anonymity is concerned, so maybe also implement basic glob or regex filters.

>>32455
>so maybe also implement basic glob or regex filters.
Filters are certainly what I was considering, but in a central censor to allow for more costly filters. Perhaps these could be togglable by the user to give a similar effect to the decentralized glob and regex filters. Obviously these posts wouldn't be federated, and would either be made hidden or removed.

The trouble is that I'm not sure the bots can do a good enough job. There are just so many ways to manipulate a conversation in a negative direction COINTELPRO - or even folks who do this work for free. I think I read that even detection of hate speech is something like a 70% accuracy affair for Facebook with much more developed tools at their disposal, and we're looking to detect far more subtle acts than hate speech.

I'm really at a loss for how to manage this, and it might be hindering my motivation to progress.

>>32461
Think there's really only two design objectives:
1. equal power to every post/poster
- no follower/following model.
- no ranking of posts by popularity.
- no names to discriminate content of users.
- (optional: following based on content via tags.)
2. quality content.
- access to a large collection of posts (federation - you could always make a new server but not you can get content on it).
- some restriction on the domain of acceptable discourse.

The fundamental contradiction is that without CIDR (centralization - and so no content) or usernames (reputation - and so unequal power) to distinguish posters I think there is no possibility of any sort of bans, and this makes creation of quality content difficult with today's level of technology.

>>32461
>a central censor to allow for more costly filters
Filters shouldn't be costly, unless there's something deeply wrong with your architecture (like emacs gnus scoring articles in a single thread by fetching them multiple times).
>The trouble is that I'm not sure the bots can do a good enough job.
They likely can't. This is why you give users the options to configure these filters, because it allows them to do aggressive filtering at their own risk. The more a filter or lack thereof bothers them personally, the more time they will spend on refining it, a textbook case of worse-is-better.

>>32471
>Filters shouldn't be costly.
Well, if we're using LLMs at all they would be from the user perspective.

>They likely can't. This is why you give users the options to configure these filters, because it allows them to do aggressive filtering at their own risk.

This tracks, it's a good idea. Having two tiers of filtering, one for basic federation (meeting basic social standards of the fediverse) and another tier with per instance filtering you could make it federate well, at least on the level of the Group. I also like the idea that sense we're working with trees of a single inReplyTo link per post you can go ahead and hide entire trees at once. Making bump ordering work with this is a tractable problem.

Went ahead and got the creation of Note and Article objects, and the display of the catalog done today with the help of "opencode". No file attachments at the moment of thread pages. Had to replace the >!spoiler!< syntax with the more contemporary ||spoiler|| so that the quotes would parse correctly.

Today had opencode add the thread posts, I didn't really like the way it structured the model code so I had to fix this.

Had another idea: a "difficult" POW for newly created accounts. This is roughly equivalent to banning everyone by default and having a default ban length. With this you could have actual user Actors instead of just the Group, but purely optional usernames, and still have bans that make some sense. Sort of think I no longer have the energy to dedicate to this project. Might just wait for the LLMs to get good enough to write this themselves.

Reincarnating this project once again.
Another cycle around the wheel.
Some interesting technical deviations.
Post-modernist feature set planned.
Skepticism of synthesizing the negated.
Leaving the unnegated pieces whole.
Designing to leave nothing behind.
Pragmatic enough to be completed.
Performance somewhat lacking.

Islands from Deno/Fresh.
Repos injected into functional cores.
Design to test, though I haven't yet.
MongoDB instead of translation or SQL.
Zod for schemas, and form validation.
A similar datatype used throughout for error reports.
Controlling for invariant and reporting errors.
Fedify worth it enough to change the whole stack.

https://codeberg.org/jugaad/anatta

Drafted a two-thirds page "spec", and stepwise "plan" for implementation of Group Person interactions. It's closer to what you'd use for a shed, than for the golden gate bridge, still way better than what have been doing. The only trouble which seems legitimate was that for threadiverse Groups to receive inReplyTos you have to Follow. The solution to this is that you can't Undo Follows, because this would break listening for inReplyTos, and so we replace it with a toggle for various visibilities. You also must Follow in order to Create(Note) on these other communities, which is not so bad.

guess tor is kill, rip; imagine how bad it would be if this site was compromised…

I didn't want to post this yet as the projects still in infancy.
But here's a somewhat topical list of features planned:
  • Follows and feeds (including for remote).
  • federated cross-posts a la >>33832 (You)
  • option for Group's which can post as the Group without login.
  • tags on Groups.
  • granular per tag moderator permissions.
  • "invisible" local karma (HN algo. sorting)
  • LLM rewriting of posts.
    • raw site with original Notes.
    • cooked site with rewritten, and click to raw view.

  • tor posting with email verified accounts.
  • websockets for liveboard.
  • pagenated CDN caching.
  • a real parser LL(\infinity).

I'm fairly happy with the feature set, am open to critique.

Have been working on the "spec" for the handle page.
It's split into a number of endpoints with several layers.
It's turned out to be relatively simple, but different than the initial implementation

  • profile.tsx
    • Initial query for Actor.
    • onScroll getCollectionService for page (in Island)
    • If the end is visible WebSockets for further data (in Island)

  • getActorService
    • Returns an Actor, and isAnonymous.
    • uses getObject, and caches to Redis.
    • CDN caches for with TTL.

  • getCollectionService (also used for replies, and Follows)
    • Returns a OrderedCollectionPage
    • Caches page data in Redis
    • "Reconstructs" the OrderedCollectionPage with cursor.
    • First page made with length modulo pageSize. (to standardize page boundaries)
    • CDN caches for with TTL.

  • postFollowService (reply and like/dislike are similar)
    • Get the necessary data from the database, and cookiePrinciple.
    • Create the Follow in DB.
    • Issue Create to recipient.



The rate limits and (granular) permissions are going to be handled by the _middleware.ts.
The rate limits are going to be by account or IP whichever is larger.

The only other big federated view is going to be the thread view which is going to use FEP-f228.

>>33832
I believe I've thought of a more elegant, if more resource heavy "spec."
Direct inReplyTos are almost always sent to the attributeTo if I understand correctly.
You then need only look up the context for each of these to get their replies.
With this you can then cache and regenerate the cache as needed with TTL.

Lemmy, and probably other instances don't issue 304 for their API endpoints, which means it has to be pummeled just to check for collection update. They also fail to use stable page boundaries, meaning you have to pummel every preceding page to get what you're after. An OrderedCollectionPage really seems like it should be organized as the N pages: {null, N-1, .. 1} with a 304, so that insertions never invalidate the cache except for the null page, and deletions only invalidate either one page or pages "to the left," if you care about fragmentation. The only downside is that queries are less efficient than when using a cursor, but this likely matters much less than not serving pages, especially when behind a CDN.

I've been writing a proxy for these OrderedCollectionPages which serves them correctly for use by the client. It still needs some work. One of my goals was to do this without actually caching any data on the server to keep memory usage low, leaving everything up to the CDN. Unfortunately the algorithm as implemented has a fatal error:
  1. We throw out all the data except for what's need for the newPage. (!)

  • In doing so we fail to update any case where updated page is less than the page we're trying to render.

  1. We allow lookupObject to cache Collections, and items, because I haven't written a DocumentLoader yet.


>>33985
Am pretty sure these constraints make the problem impossible…
Guess I'll deal with it later.

I'm still working on this algorithm. I'm trying my hand at formal specification.
The head of the algorithm is given as follows:

  • Pages are indexed as 1..N-1, Sentinel so that 1 is the oldest posts.
  • Page boundaries are completely stable.
  • We lazily load the item statuses, finishing if we've fetched the cached page or all the updates.
  • If there's no changes and we're fetching the end we skip to the end.


The Init and Next formula follow:

Init ==
  /\ TypeOK
  (* Insures "new" statuses are only in continuous ranges connected to the *)
  (* ends. *)
  /\ NewAtEndsOnly
  (* Set item and page to either jump to next or start at Sentinel *)
  /\ JumpToNext \/ GoToStart
  (* We do not assume a consistent pageSize for pages. *)
  (* We do not assume that pages are defragmented of a certain number *)
  /\ PageItemsMonotonicalIncreasing 
  /\ PagesLessThanPageSize
  (* statuses is a record of the state with respect to the pages cache. *)
  /\ NewItemsNotInPages
  /\ DeletedItemsInPages
  /\ UnmodifiedItemsInPages
  /\ changes = 0

Next ==
  \/ /\ statuses[item] = "new"
     (* If we've read all the changes needed we stop updating. *)
     /\ changes /= netChanges \/ ~Cached(desired)
     /\ pages' = [ pages EXCEPT ![page] = pages[page] \cup { item } ]
     /\ IF Cached(page)
        THEN changes' = changes + 1
        ELSE changes' = changes
     /\ Iterate
  \/ /\ statuses[item] = "deleted"
     /\ changes /= netChanges \/ ~Cached(desired)
     /\ pages' = [ pages EXCEPT ![page] = pages[page] \ { item } ]
     /\ IF Cached(page)
        THEN changes' = changes - 1
        ELSE changes' = changes
     /\ Iterate
  \/ /\ statuses[item] = "unmodified"
     /\ changes /= netChanges \/ ~Cached(desired)
     /\ pages' = pages
     /\ changes' = changes
     /\ Iterate

Think managed to get the math to check out.
Replaced the page variable with CHOOSE which picks the best location for the item.
And added a Init constraint that EmptyNewPages s.t. there are pages available for all the new items.
Had to replace most of the initial type declarations.
Did this to allow arbitrary subsets of Items, and because modulo for Sentinel size was no longer working.
Still not sure it's perfect.

apparently vichan has a bunch of 0days that haven't been disclosed yet but can give an attacker a shell on the host. considering this site is a fork of vichan, are there plans to move to something else? this was last month, and the vichan guy on github decided to archive the repo instead of working on finding the vulnerabilities and fixing them

and yes, this sounds important I'm too lazy to make a new thread


>>34079
>>34080
uhh this seems bad

File: 1792416642821-3.png (433.99 KB, 480x480, ClipboardImage.png)

Wasn't going to post this yet, but because the thread got bumped. Deno/Fresh got nuked by Cloudflare (evil), and I realized using MongoDB was kinda stupid when Postgres works perfectly fine as a document store for many years now. Even React/Preact is sort of stupid to use in an app that has virtually zero interactive components outside of HTML forms. Still need an island architecture to avoid SPA but seems like the only option is Astro which I don't really like because it makes you use .astro files for this.

Another day another tech stack… (retro-future) 90s edition.
  • HTMX
  • typed-htmx
  • pg
  • express
  • express-file-routing
  • Node.js
  • fedify


Am probably going to preserve the architecture I already have in the Deno edition. Just porting everything over. So far just have one day of work to get everything initialized so that I can start the port work.

>>34608
why node of all things. why not stick with the boring choices i.e. jvm or dotnet languages or even golang/rust. Hell even python.

Why server side javascript in 2026?

>>34612
Was reading this ad roll about Netscape LiveWire Pro Web https://philip.greenspun.com/wtr/livewire.html it wasn't perfect but I thought to myself, eventually everyone will know it because Netscape is a big mover, and maybe the libraries will come along, and enough will be invested in it to make the runtime fast, and you know maybe we find ways to gloss over the troubles with the language given time.

I know some people subscribe to this idea of "innovation tokens" or some similar market language. But I really think we can go fast, The future's at our fingertips, if only we're willing to take it! "The club hits of today will be the show tunes of tomorrow."

Day 6, thought would give an update. Whole thing so far has been vibe-mageddon. Have been reviewing, and understanding, and it's largely a port of the Deno version to a different tech stack, but also exhausted all my free tokens yesterday.

The basic architecture is pgtyped SQL generating repos wrapped in a (railway) error handler wrapped in a (unit of work) transaction and (dependency injected) passed into the service layer from the file routes. Basically the same as the Deno version but with SQL instead of MongoDB etc.

We're at something like 3000 lines now. I think I've got the database mostly implemented, all the architectural plumbing, a good chunk of the components, basic federation, and the API designed (but not yet implemented). In terms of what's actually working at the moment it's just session management, and user federation but much of the supporting structures are in place for the rest.

No git hosting yet because my platforms, quite wisely, have AI bans. I'll probably have to self host the git instance and throttle.

Let things get a little out of hand as the port work was/is finishing up. So had to do a layer by layer, and file by file review and edit by hand. Nearing feature parity now with my original implementation posting is working as are profiles, including posting to anonymous groups, both as the group and as a user. The review is just about complete barring two files to do with the federation layer where some more comprehensive edits are required and cross-referencing with the documentation. IIRC what's left for feature parity is the follow page, and the thread pages. Should be pretty simple.

Reached feature parity with anatta-deno, and completed the hand review yesterday. At 4500 lines now, am hoping the whole thing, including tests will be under 10000 - so at least 45% done. Next big feature is the feeds and sorting algorithms (HN, Bump, and Recent). Only other huge feature can think of is adding the doomscoll/pagination everywhere. Then there are a ton of smaller features.

Just completed feeds, and sorting, called them Score, Bump, and New. 5000 lines now.

Am thinking of something sort of complex to avoid sybil attacks:
  • A minimum karma statistic for votes, and posts to count towards rank.
  • SybilGuard once passed the karma statistic to check for voting rings.
  • Escalating bans from board if sufficient rank changing negative karma (broken hearts).


So even if you had a residential proxy service with many IPv4s, and an email server you couldn't manipulate.

Day 14 update, 6000 lines total. Got 99% of pagination & doomscrolling working, a small error to do with posting that will fix later. Got "remember me" working, a parser for posts, and avoid relay behavior in the federation layer. Plan for the day is finish off pagination, add a captcha for signup (the only captcha), and HTML sanitation for the federation layer.

>>34731
Got all this done yesterday. Plus started on remote profile fetches — they aren't currently signed so some of the fediverse 404s. We're probably looking at another two weeks until feature complete. Have also been working on language enhancers this may as well serve as an all call for anyone interested in sharing what they'd see changed in English. I've got a pretty long list but I'll leave it as a surprise, mostly they seek to subjectivize, relax, and encourage flow. In a separate all call will soon be interested in recruiting UX that is creative or particularly good from you all. Assuming anyone expresses interest.

I'm really starting to blow some of my complexity budget on caching and fetching of remote profile outboxes. About 1200 lines. I'm ripping it up, and just having remote profiles contain messages that have reached the federation endpoints via follows, and replies. Apparently this is what mastodon does anyway.

Am going to come back down to earth a little, and stop complecting, the SybilGuard is likely to be the most complex part of the application (but seems necessary). Next steps after this are votes and deletes.

>>34749
Yi xin edition. Got some buttons up, still have to review the karma code, and implement the delete/report button. Good news is that bailing on the outbox traversal, and now on the remote Likes the only thing left in the federation layer is fep-228, and posts addressing multiple Groups (both of which truthfully could wait).

Karma system, and Delete/Tombstone completed, reviewed, and committed. Also reworked the git history using lazygit and a lot of care (a bit too late), so that some of the mistakes are a little less obvious. Line count 8500. Plan for the remainder of the day is the Follow code, and maybe the Settings. If we're lucky might even touch the Group creation page.

>>34760
Follows, FEP-228, and settings complete 10000 lines total… Next is a (likely) massive amount of moderation code. After this there's only a handful of features levels left: LLM, rate limits, email verification, and style. Well and all the tests (yes it's terrible that I haven't been testing all along). I'm aiming for under 20k, and never a large project >100k. Once all this is done now have a two week review of the whole thing planned.

>>34770
Project size increase by 60% for the naive moderation system. 16000 lines now — though some of this is pgtyped generated. Thought it would take three days to review, but its seeming now like it's going to take longer.

Acquired a domain today: anatta.social (nothing's there atm), set up cloudflare, and made the source code available at https://gitlab.com/anatta/anatta (chose gitlab because it's likely to stay up, allows AI, and is mostly FOSS) up to 19500 lines just from small edits here and there. Projections at 30k for the whole thing now, including tests, and three more weeks at least.

>>34821
Perhaps 80% through the fairly weak review of the moderation system, SLOC has been fluctuating but at 20000 now. I'm going to put a pause on the rate limits and LLM censor for unit and integration tests after this. Hopefully tomorrow I'll be able to finish the review up, without rushing too badly.

>>34837
Bad news, set the LLM in the night to generate a test suite, file by file, aiming for conditional coverage, with unit tests for everything but the routes and integration tests for those. And it produced 27k SLOC of tests, almost all of which are passing… So they're probably massively over-fitted. I've instructed the LLM to attempt to revise the test suites with the same constraints but attempting to generate more failing tests. In the meantime I'm going to attempt to finish off the last 20% of the review of the moderation system.


Unique IPs: 9

[Return][Go to top] [Catalog] | [Home][Post a Reply]
Delete Post [ ]
[ home / rules / faq / search ] [ overboard / sfw / alt ] [ leftypol / edu / labor / siberia / lgbt / latam / hobby / tech / games / anime / music / draw / AKM / ufo / 420 ] [ meta ] [ wiki / shop / tv / tiktok / twitter / patreon ] [ GET / ref / marx / booru ]