[ overboard / sfw / alt / cytube] [ leftypol / siberia / hobby / tech / edu / games / anime / music ] [ meta / roulette ] [ GET / ref / booru]

/tech/ - Technology

"Technology reveals the active relation of man to nature"
Name
Email
Subject
Comment
Flag
File
Embed
Password (For file deletion.)

New Announcement: IRC<=>Matrix bridge #leftypol on Rizon
Feedback Wanted! : Designing Transparency by Default
Proposals done until Monday : /meta/


File: 1631336482854.png (261.26 KB, 480x481, God.png)

 No.11160[View All]

Every few months, people come up with an idea to create a new imageboard, since mostly everyone is dissatisfied with the state of vichan/lainchan, or thinks they can do it better. Having better software would greatly enhance the experience for both regular users and mods, and have more crossover appeal to “normies”. The problem is that no one can agree on the technical or more importantly non technical decisions on how one would go about making an actual, usable replacement for lainchan.

In fact, people don't even agree on whether the imageboard replacement should be an imageboard at all. This thread is a merged, consolidated, megathread of all the various attempts at answering this question that people have made.

Previously, there was a thread on outreach to lainchan, including a strawpoll:
https://lainchan.org/%CE%BB/res/26674.html

The poll determined it should be built in Java, but a significant minority wanted to use a functional programming language, esp. Haskell, or Clojure/Lisp.

The only way a new imageboard will be built is if multiple people, technical jannies and lurker-programmers, from here and lainchan and even elsewhere, actually collaborate on a single project and concentrate their efforts on this.

There are important technical and non technical questions to be answered.

Namely, do we even want an imageboard?

Some anons have suggested a radical re-imagining of the medium of anonymous communication where boards are replaced with tags. Some want named accounts, some want textboards, the ideas are endless. Is there anything to be improved upon with the imageboard concept, and if so, what? If reddit is an improved version of the phpbb type forums of old, what does a “modern” version of a chan look like?

Feel free to use this thread for any polls, discussion, of the formats, technical, or non technical decisions related to a vichan replacement both for this site and in general.

Also comment on:

Concept: Classic Imageboard vs something else
Architecture: Monolith vs Microservices
Distributed: Distributed/Federated vs Standalone
Front end: SPA (ex: Angular, React) vs Server Side HTML templating
Backend Language: Java, C#, Lisp, Rust, Golang, PHP, etc.
Database: SQL vs NoSQL
325 posts and 51 image replies omitted. Click reply to view.

 No.11486

File: 1630630715043.png (234.96 KB, 351x430, animesmoking2.png)

>>11484
>You have to go an manually check for replies, which is further exacerbated if you use tor browser (which you should) and have to remember every thread you replied to and check them individually.
Notifications can't be implemented without user accounts, which goes against the low barrier to entry of imageboards. I wouldn't be opposed to making a chan that required registration but still showed users as anonymous publically.

As for distributed imageboard systems, some have been made but the technical issues would actually change the nature of what imageboards actually are as discussed in >>11464

Like the post said, anyone who talks of making a "distributed" imageboard needs to explain how it would deal with these basic issues of how site mechanics would work or translate from traditional chans to a distributed paradigm

>>11485
yeah but that doesn't explain why reddit is popular compared to chans, since it's mostly the same in that regard. The only exception is the occasional AMA

 No.11487

>>11484
You are correct about a lot of things but a whole new protocol would only push further into the obscure corners.

 No.11488

>>11486
>Notifications can't be implemented without user accounts
They absolutely can

 No.11489

>>11488
Yeah by storing everything in session data, along with themes etc. in which case you’re halfway to defacto implementing accounts anyway

 No.11490

File: 1630731968959.png (253.54 KB, 351x430, pointing.png)

>>11489
As stated, if you're going to start storing tons of data in session/cookies to make up for lack of accounts, you're halfway to de facto implementing accounts without calling it that, with significant technical overhead needed to track all this stuff, all so people can avoid having to press the login button. A better solution is to implement accounts but allow people to comment anonymously as always, however, put additional features on accounts like notifications, saving preferred theme between sessions, maybe direct messaging, hell you could even get more accurate metrics on the true amount of users. It could also help during raids if you disable anonymous posting temporarily. Also you could require TOR users to use an account that way you don't just have to resort to banning the entire TOR node, and can allow legitimate users to continue using it. OFC the spammer could always create another account, but nothing's perfect.

CAPTCHAs and other anti spam measures could be pushed off to the account creation stage, leading to better UX for the user instead of making them solve a captcha any time they want to make a thread or w/e.

In fact a ton of the spam problems of chans/imageboards could be solved with accounts, merely by having them as an option.

 No.11491

>>11484
new protocols for everything is a complete and utter waste of time.
to get more people using imageboards the number one thing to do is identify areas where a discussion space is needed and then create one using imageboard software, without carrying the stupid baggage of chan culture with you, and then trying to create a board that best meets the need of the resulting community.
imageboards are no more difficult to use than newspaper comment sections in principle, the problem is purely one of application. fundamentally there's no reason you couldn't use lainchan to host a website for old women to discuss knitting patterns. sure, they'd probably actually enter their name and email (oh no!) unless you got rid of those fields, but you could certainly do it if for whatever reason you had contact with enough old women to get it started.

a new protocol wouldn't help. an imageboard with its own protocol is either something to ignore or (if by some miracle it actually had enough users to be worth caring about) something to access via a browser addon that the scum at mozilla will keep intermittently breaking on purpose because they hate all that is good and holy and know that they can get by with being marginally less evil. browsers are their own hellscape, but "make a new protocol" is a false dawn for breaking out of that. at a push, it works for large tech companies who're probably going to have the application to access their protocol just be electron reskinned-chromium bloatware anyway.

 No.11492

>>11490
>implement accounts but allow people to comment anonymously as always
How about enforcing anonymity?

 No.11493

>>11450
discoverability mainly

 No.11494

>>11489
>>11490
What a sad state of things we have in 2021 that people don't even think about implementing stuff outside the browser…
>>11491
> something to access via a browser addon
Again, why?
>browsers are their own hellscape, but "make a new protocol" is a false dawn for breaking out of that.
Ok what other options are there? By your own protocol I don't mean stuff like freenet or bittorrent, but at least a standardized json api for different server software (presentation layer).

 No.11495

>>11487
>>11491

The gemini people made a new web-protocol and it's working for them
maybe the trick is to make it very simple so that it's easy to maintain ?

 No.11496

>Notifications cannot be implemented without identification.
The client would need to keep a post database and periodically poll the server. You could implement a watch feature that enabled this for a thread, if it was unfeasible for all posts.

 No.11497

>>11492
you mean like blocking namefagging or not even allowing it? sounds good to me but on the old /get/ on .org when it was running different software all the /get/ guys just namefagged in the comment/body section anyway

 No.11498

File: 1630744103324.png (122.28 KB, 233x533, 80s_chick.png)

>>11497
yeah, if you dont allow namefagging all the namefags will just sign their name with
-le namefag

boomer email style.

 No.11499

>>11496
yeah but then thats multiplying the number of get requests/polling even more than the normal amount for refreshing threads, recall that is what crashed lynxchan. I mean i guess it could be done, it would just be more resource intensive than not doing it

 No.11500

>>11499
If this was part of a new protocol, it could be done over a UDP port. As this would be an optional feature, TORfags won't complain.

 No.11501

>>11500
alot of ISPs block UDP packets nowadays, so it would probably have to be TCP.

 No.11502

>>11500
i2p has something like udp iirc

 No.11503

reject notifications and hyperonline 24/7 cyberspace-injected-into-eyeballs annoying shit.
return to ancient unmaintained imageboard PHP autism via tor browser.
it's a natural newfag filter, and anything else is destined to become surveillance capitalism

 No.11504

>>11486
>a chan that required registration
That's heresy, that would be literally reddit.

 No.11505

If there are (you)'s it's perfectly possible to get notifications, I think that 4cucks already had that.

 No.11506

File: 1630877775295.jpg (38.11 KB, 750x503, relative.jpg)

>>11504
No it isn't. You lose some benefits and issues of the anonymous imageboard, but it's not like reddit at all. A forum, maybe, but not even.
Look at 4kev for an example.

 No.11507

File: 1630877822039.png (17.07 KB, 500x217, ClipboardImage.png)

>>11505
We have it here, if you enable it.

 No.11509

>>11507
uyghaaaaaa what. Ive been relying on control+f (you) for so long.

 No.11510

>>11509
doesnt work for me

 No.11511

File: 1631358548507.png (33.15 KB, 381x521, apparch.png)

I've been thinking if leftypol does go to a more sophisticated architecture it should store all images and binary files in s3.

Whats everyone think of this setup?

 No.11512

>>11511
A perfect backend would be storage agnostic.

abstract class StorageProvide
FileStorage
S3Storage

Not sure what others make sense. Amazon is not cheap, and requires some hoops to get an account. So requiring people to use Amazon's services might be a non starter.

 No.11513

>>11512
most cloud hosters provide an S3 compatible object store, even low tier ones like digital ocean and OVH.

Even if they don't most open source object store software like minio, ceph, riak, etc. uses or at least provides an s3 compatible API. The only reason corporations don't use minio is because the free version is AGPL licensed, but thats not a problem for us

 No.11514

>>11512
>storage agnostic backend
Why not access all data through the filesystem. The hierarchy could be backed by a database over fuse or 9p with software like https://github.com/greenbender/sqlfs

 No.11521

>>11514
because what if you have multiple app servers that need to share images, etc.?

 No.11522

>>11521
That depends on what filesystem you use and does not need to be tied to any block device. You can already mount s3 as a filesystem https://pypi.org/project/s3fs/

 No.11527

>>11514
what exactly is the use case for this? why do you need an encrypted "mostly read online" filesystem for an imageboard?

 No.11528

File: 1631412743415.png (270.12 KB, 1662x2055, Database ERD.png)

I've been looking at gochans schema is it is way too complicated. Also judging by several of the field names it violates some normalization rules and rules of thumb.

You're not supposed to have "derived properties" like is_top_post. A top post can be determined by lowest id or lowest create date. Therefore having it as a field is a redundancy. Same shit for "file order". You should insert the files in the proper order and then order by id. there's no need for a "file order" field.

Some of the complicatedness seems to be supporting some auditing features that would probably be better of in a log.

Sorry rat. I don't approve.

 No.11530

>>11461

that's called P2Pchan and, like most of tee's other attempts to make something interesting/stable/usable, it sucks.

 No.11531

>>11527
My bad, I was looking for something like https://docs.microsoft.com/en-us/sql/relational-databases/blob/filetables-sql-server
Regardless, the server could store files on any filesystem, including ones that directly map onto an existing database format.

 No.11532

>>11531
so its just storing binary blobs in the database but with streaming so you can start playing it right away? I don't know if its worth it the only reason to keep files in the DB like that is if you need them to be transaction safe and have ACId guarantees or w/e. I don't think shitposts really need that much integrity

 No.11533

>>11532
If you wanted to eschew transaction safety, you could store the data on a network share (NFS, SMB, or a 9P files server), or a regular filesystem while a daemon synchronizes multiple servers (using rsync, or a VCS over ssh).
Using the filesystem as the storage interface of the server allows it to access anything you can mount.

 No.11534

>>11533
rsync could be ok if you have 2 servers, id hate to scale that to 3-4 app servers. There's some tool that uses rsync to do live real time updates as a daemon, I forget the name though.

I've always had problems mounting storage from multiple servers.

 No.11538

File: 1631499415035.png (149.95 KB, 329x253, smoking.png)

The essence of a chan

This is a followup to the “ideal social media” post. I’ve been thinking about site mechanics and how they influence behavior. I’m trying to drill down on what a chan is, really at it’s core. In silicon valley, they often do these “[site concept] in one SQL statement” type overviews.

To analyze a site/app we have to think about the behavior it “mechanically incentivizes” i.e. how the actual mechanics of the site/app force or incentivize the users to do certain things and behave in certain ways. The essence of reddit is the link. Reddit is a link aggregator which means the whole site revolves around the posting of links. In this sense the very behavior that reddit is so hated for, stealing OC, is literally the basis of the site. The basis of twitter is saying shit people like. On twitter, people are rewarded with likes. Having a post with more replies than likes is called “getting ratio’d” and is considered bad. However on a chan the incentive is the literal opposite, more replies mean the post is bumped to the top. So “taking the bait” is literally built into the mechanics of a chan. While the twitter way encourages a “Los Angeles Hollywood” style mentality, of performative takes to get likes and followers, doing and saying things you think other people will like, chans mechanically incentivize saying things people will respond to. Also chans have a group identity based on boards or the site itself.

chans/imageboards are inherently filesharing sites. They revolve around the sharing of files. But the files themselves are small, and often original content. Unlike, say the pirate bay which concentrates on sharing large files of decidedly not original content (movies, etc).

Every OP on an imageboard is essentially a file or collection of files, with an optional comment, and some metadata for discoverability. What separates it from something like instagram is the anonymous nature of imageboards, transient nature, and lack of personal identity.

So a chan must:

• Be Anonymous
• Emphasize activity/responses over popularity
• Emphasize originality over derivative content
• Emphasize group identity over individual clout
• Be filesharing-centric
• Be Transient

In order to discover the true essence of a chan we must ask ourselves, what if every aspect of this was hyper exaggerated to the max.

Anonymous – Anonymity serves several purposes. Contrary to popular belief, “free speech” is not one of them. While free speech is good, it’s not inherent to a chan, nor are non-chans inherently unfree. What speech is allowed is a consequence of the moderation of the site, and you can check various alt-tech platforms or dis.cords to find content that’s just as edgy as that of a chan, if not more. What anonymity does do, is prevent individuals from accumulating clout (or money) on the basis of their statements. Since everyone is anonymous, and there is no “follow” or “subscribe” function, this means users are not mechanically incentivized to say things they don’t mean, just for the likes/followers. Everyone is equal, everyone is just an anon. The main value of anonymity isn’t free speech, it’s anti-celebrity. In order to be even more anonymous, all namefagging and tripfagging should not be allowed, in fact the name field, email field, and tripcodes should be totally eliminated. Additionally (you)s should also not be allowed. Notifications are fine but should simply state that there are new replies in a thread you posted in, not if they specifically replied to your post.

Emphasize Activity – One of the most neglected aspects of site/app mechanics, but most important is bumping, or the order in which posts and threads are displayed. Twitter displays tweets by creation date of the OP. This works well in incentivizing people to make new threads because old tweets get left behind and people lose interest in them. However, the incentives on chans are to make threads that get a lot of replies. Twitter incentivizes more content, but Chans incentivize more triggering content. Due to the catalog limit, this leads to a sort of collaborative sorting, a brutal darwinian ecosystem in which the boring posts sink to the bottom. I’m not sure which is better, twitter create date sorting or sorting by replies. Sorting by last reply incentivizes edgy posting, but also baiting. Also cyclical threads just end up become glorified chatrooms, sometimes with namefags to boot. Are “generals” really a good thing? Or should people just create a new thread for every single topic. If people are still stuck on the “a thread died for this” they can just increase the catalog size from 350 to 500 or something. Or change the formula to a time-based formula, like have posts last 2 weeks.

Reddit uses a more complicated formula for whats displayed, including posting time decay formulas, likes, etc. I don’t think a more complicated formula would be good.

Maybe its better to stick with the last bumped formula, but twitter style has its advantages too. However ordering by create date of OP would mean that shit like saging, cycling, bumplock, etc. would essentially be obsolete since posts would not be bumped at all.

Originality – R9K enforced on all non archived posts, both imagehashes and text/comment. Literally to the point of putting a unique key constraint enforcement at the database level. In addition to showing number of replies, images, etc. threads should also count/show the number of effortposts, meaning 1400 or 1500+ character posts in a thread. Autoarchiving of all threads, as a setting per board. So for example autoarchiving could be turned off on /b/.

Front page would be overboard.

 No.11540

>>11538
>all namefagging and tripfagging should not be allowed
Perhaps people would start to sign their posts or pgpfag, but it allows for more simple post storage.
>Are “generals” really a good thing?
Generals have plenty of activity, so maybe they should be substituted for their own if slow board. With hierarchical boards, country specific boards would not clutter the top directory.
>putting a unique key constraint enforcement at the database level
The server could store data on something like a venti filesystem http://doc.cat-v.org/plan_9/4th_edition/papers/venti/
Posts could also just reference files by a hash and upload them afterwards, if they don't already exist, allowing to seperate both actions without identification. Tags should be enforced on file upload, so the server may act as a booru and prevent image duplication.

 No.11541

>>11540
>Posts could also just reference files by a hash and upload them afterwards, if they don't already exist, allowing to seperate both actions without identification. Tags should be enforced on file upload, so the server may act as a booru and prevent image duplication.

AFAIK wordpress operates similarly in that you can re-use uploaded images between posts. It sounds good but the only problem I can see is that people might be too lazy to upload files separately.

Basically you would be combining the functionality of a booru and a regular chan

 No.11542

File: 1631516428649.png (417.32 KB, 631x472, phone.png)

I've also been thinking about boards and if they're really necessary. Yes there is a board identity. However on small alt chans there really isn't enough traffic to justify multiple boards. Besides, you might want to make one post which is on an unrelated topic without having to justify a whole board for it. So IMO tags are a better option than boards, for alt chans specifically.

That way if you have 300 posts on one topic and one unrelated post it can just live in a different tag rather than either having to delete it as offtopic or make a whole new board for it, or shove it in some sort of /b/ or off topic/general type board. Boards seem to be shoehorning content, unless you have a major traffic site like 4chan which justifies that sort of silo-ing of content. Of course the topic of boards also leads to the idea of catalog size. If there are no boards how do we determine what to prune? And if posts can have multiple tags, under what category do they get pruned?

IMO a better solution would be to prune threads which get no replies after a certain time period, say a week, regardless of tag, unless they have a special status like cycle/pinned, etc.

 No.11543

>>11541
The seperation of posts and images would be at the protocol layer.
It could be made invisible at upload, essentially looking the same as vichan, except for the tagging and faster upload, when a file is already present.

 No.11544

>>11543
hows that qualitatively different from having a files table in a database with hash checking for uniqueness though? most chans already have that as optional

 No.11545

>>11544
Non-malicious clients will transfer less data.

 No.11546

Potential impact of different thread grouping:

<board based

>either slow boards or less discoverability for niche topics
>off-topic boards incentivize shitposting
>many slow boards clutter the board list
>an overboard may be created for low activity boards

<hierarchical board based

>slow boards would be a non-problem because every directory is an overboard
>still allows for comfy boards or directories
>pruning could be set at depths or individual directories

<tag based

>may incentivize forum-like topic discussion
>may lead to less discoverability for shitposts and niche topics
>pruning by last reply date kills slow threads
>no board identity

 No.11547

>>11546
>may lead to less discoverability for shitposts and niche topics
I mean they will still show up on the overboard, right? Plus you could have multiple tags so have them in one mainstream tag and whatever niche tags they want.

Hierarchical board based is an interesting idea, the SQL for showing it could get pretty complex though.

you should explain hierarchical boards more though

 No.11548

could the hierarchical concept be combined with tags and create a hierarchy of tags?

Also how many layers deep would the hierarchy be? I'd hate for the app/user to have to parse a tag/board tree N layers deep every time they want all the posts for some broad category of things. And could users create these tags or would it be janny based?

 No.11549

File: 1631527695523.png (208.07 KB, 351x400, animesmoking.png)

I guess my problem is every time you add a subcategory of a subcategory, you have to join the categories table to itself, and joins are expensive. If you're 20 subcategories deep that would lead to extremely bad performance, at least from a traditional relational DB. You want to minimize the amount of joins. Tags are flat, just one join to the tags table and you can get whatever you need.

IMO the first step is decoupling the pruning logic from the tags/categories discussion. That way we can consider it in terms of discoverability, and have pruning based on last reply+1 week, not some hard catalog limit like 350 threads. This would also eliminate damage from sliding and raids because creating a new thread would not be a destructive act (deleting old one).

Although the catalog view would have to be rethunk, because if the amount of threads can be arbitrary, the catalog can only ever show the top of them.

There's also the pub-sub/group model where users can subscribe to certain groups/topics/boards and their timeline would be an amalgamation of all of the boards they are subbed to.

 No.11550

>>11547
>you should explain hierarchical boards more though
To detail my approach to a hierarchical imageboard:

I'm working on a non-standard http server that translates POST requests to writes instead of CGI execution. By default the server would have read, write and exec permission on directories, read permission on all files and directories would be created sticky.

Mounted at the root of the server would be a filesystem, that passes through all writes and on reads generates static pages on the fly (or with some form of caching, idk yet).
Boards would be directories not starting with a special character stripped by the client, otherwise they would be threads.
Either posts would reside as plaintext files and have a header with file references. In this case the client would write to that file.
Otherwise a post would be a directory without write permission, with a plaintext body file and fields or references as separate files, which means it could not be directly written by the server, instead being written as an asynchronous request to a fifo, which has higher throughput and is more secure than a CGI request.

>the SQL for showing it could get pretty complex

I hope this makes clear that no tables are involved in the hierarchy. Administrative tasks would be handled by communicating with daemons through fifos or direct file/directory modification.
For bumping, threads would always be named by their last reply and the catalog would be like 'ls' in descending order. Anchoring and pinning could be done by adding special symbols to the thread name.

>>11548
>could the hierarchical concept be combined with tags and create a hierarchy of tags?
I guess the motivation for this is, that tags are more flexible than hierarchies, yet this may be corrected with symlinks, potentially created by a request over a fifo.
>I'd hate for the app/user to have to parse a tag/board tree N layers deep
Only a client directly accessing the file hierarchy would suffer from this. A special-purpose IB 9P server could be written, where mounting an overboard compiled a list of threads with limits set by the client.
>could users create these tags or would it be janny based?
User creation of directories would be ideal, but small servers may not want this and would set file permissions accordingly.

>>11549
>decoupling the pruning logic from the tags/categories
IIRC the main motivation for pruning the catalog was storage constraints.
Because storage availability has changed, you have the right approach, that is probably to prevent necrobumping, and it could be implemented by a daemon deleting based on directory modification time, perhaps even inactive board hierarchies.

 No.11552

>>11550
Interesting ideas although I was looking for something more in line with a traditional web app. Good luck with your project though, keep us all updated.

Also what do you think of the trade offs of ordering by last reply vs Twitter style ordering by create date with most recent first


Unique IPs: 13

[Return][Go to top] [Catalog] | [Home][Post a Reply]
Delete Post [ ]
[ overboard / sfw / alt / cytube] [ leftypol / siberia / hobby / tech / edu / games / anime / music ] [ meta / roulette ] [ GET / ref / booru]