[ home / rules / faq ] [ overboard / sfw / alt ] [ leftypol / edu / labor / siberia / lgbt / latam / hobby / tech / games / anime / music / draw / AKM ] [ meta ] [ wiki / shop / tv / tiktok / twitter / patreon ] [ GET / ref / marx / booru ]

/tech/ - Technology

"Technology reveals the active relation of man to nature" - Karl Marx
Name
Options
Subject
Comment
Flag
File
Embed
Password(For file deletion.)

Not reporting is bourgeois


File: 1726459786963.png (365.18 KB, 709x538, nuimageboard.png)

 

The neverending quest to rewrite vichan -

Archived threads:
https://archive.is/xiA7y
115 posts and 38 image replies omitted.

>>26999
>but this may have to wait for another day.
Nope, haven't fully tested it yet but believe the whole board is live now, including moderator actions, and the catalog, the latter of which may be a little chaotic.

>>26997
Got your feature request implemented also. This is a page with the optional search parameters set such that the white theme is used (one of my favorites), with no (you)s, and autorefresh turned off.

Trying to decide between implementing email verification (so users register other's emails) and anonymous users (together with address based logging, flood control, and bans). Probably going to try my hand at the latter first since it doesn't involve learning any new libraries or complicated configuration. Is Shii right? https://wakaba.c3.cx/shii/shiichan

>>27004
Got Anonymous posting, address based logging, flood control, and bans implemented. Would like to have it where it was a per-board configuration option for Anonymous posting, but don't see how it could be easily implemented at the moment.

>with vichan every theme, except the default and yotsuba b, has a noticeable delay for every page, so you might want to verify if themes get cached and if there is significant latency during loading.

Have been trying to solve a similar issue to this where there are a few milliseconds of unstyled content which flash when using the theme selector. Apparently the only way to fix this is to modify things server side to work with the search parameter.

>>27007
>Would like to have it where it was a per-board configuration option for Anonymous posting, but don't see how it could be easily implemented at the moment.
This is now implemented. Source including the frontend is now at around two thousand line. The whole thing probably needs a once over, and a lot of usage to find any missing bugs and inconveniences. Here's a list of my todos at the moment:

Clerical
- host source
- public instance
- project name (presently "cyberia")

Source
- word filters
- captcha (after some number of failed login attempts, and on signup)
- email verification or validation
- reimplement themes in backend
- views for user and address reports

Am currently implementing the captcha system. Might finish it tonight. Think could get away without email verification if this was done right. There's currently no plans for range bans, that might be a mistake, wonder if they're used often in practice.

>>27008

Eh, might scrap the captcha entirely just to use timeouts. Got one setup for the login screen, just need to copy paste and edit a little for signup.

>>27010
Probably added timeouts for signup now.

Made the quickreply work on mobile. The mobile site and the sockets seem like real improvements over vichan.

>>27013
Got previews working on mobile, it's hold to preview, tapping anywhere then closes the preview, and normal tap follows references. Added user-select: none; to the message headers, nav, and quick-reply forms, and references to prevent accidental interactions. Also cleaned up the mobile quick reply a little more so that the page is always big enough to hold it. Attached is a picture of what this all looks like.

>>27015
Please ignore the bug in the photo.

The last thing on my todo list as far as software goes is "views for ban reports.", which is going to be put off for now. Probably what it needs more than anything is testing and polish or bug fixes.

Would there be interest in using an LLM for moderation (it'd probably require quite the machine to run)? Or would you all consider this to be an anti-feature. It doesn't look like it would be that bad to implement.

File: 1730908472560.png (1.05 KB, 48x48, favicon.png)

>>27018
>it'd probably require quite the machine to run)
Found a tutorial to use Llama Guard, it seems pretty easy, but it suggested and APU with 16Gb of ram which is more than my server would have. Also more than most small imageboards would be likely to have. Might end up playing around with it anyway. Also made a small favicon for cyberia, and decided to stick with the name for now.

>>27022
Been mostly fixing bugs today. Looks like there's a 1 billion parameterized Llama model with quantization. The memory usage is at about two gigabytes and it runs even on my 2013 dual-core without too much trouble (though it definitely requires swap because Mac OS is not a low memory OS). Think this could be a neat optional feature. What do you all think about this? A link about what it's attempting to censor: https://huggingface.co/meta-llama/Llama-Guard-3-1B#hazard-taxonomy-and-policy

>>27025
Yesterday gave the whole program a read through correcting errors along the way. Also expanded on the todos a little. Managed this morning to get thumbnail resizing setup to save on bandwidth. Would like to setup incremental autorefresh next, but don't exactly know how to do it without dropping messages, especially considering there are bulk operations.

Clerical
- host source
- public instance

Backend
- views for ban reports.
- easier way to create the admin account.
- LLM for moderation. (Llama Guard)
- prettier error pages.
- incremental autorefresh

>>27030
Styled the error pages, cleaned up the error handling in source, and add proper handling for 500 errors in the mean time. The incremental refresh is turning out to be quite tricky. Was thinking could take a hash of the page to check for dropped messages, and just refresh the whole thing like is currently always done in this case, but it turns out the preview script modifies the DOM and this is no longer a reliable measure. Am open to suggestions on how of if its necessary to do this. Is the gevent loop just fast enough? The LLM has to wait for me to get a new computer to avoid swapping.

>>27030
The only addition to the program after this morning was the simpler system to create admin accounts. Realized to setup incremental autorefresh would require reparsing posts to generate backlinks and at that point gave up for now. The view for reports of posts, users, and addresses requires me to make a query that don't know how to express in SQLAlchemy think it would be an outer join, and coalesce. The LLM requires me to have a better computer to avoid swapping while developing. So that's it from me for now. Releasing the source is the only thing which seems possible for now, and still haven't decided on a platform for this.

Open to any suggestions.

>>27018
I suggest you add the ability to run messages through an arbitrary server-side spam detection program. See https://bogofilter.sourceforge.io for something like spamassassin, that also works for plaintext.

Site doesn't even work yet but already has a link at top to beg for money.

>>27045
>ability to run messages through an arbitrary server-side spam detection program.
Added, though this is a feature for advanced users given the implementation. Will use this for my LLM moderation system and have it as a separate program.

>>27045
>beg for money.
In the lupen sense and not the grifter, went ahead and removed it from the temporary nav. Not a clue what boards to actually include either if it ever gets hosted.

Also added stickies and locked stickies, styled the success pages, and cleaned up the error reporting some more.

>>27046
Added support for the remaining MIME types today, so video, audio, and documents. This included making thumbnails for PDFs, and having a fallback thumbnail for documents which don't have their own (the document formats other than PDF).

Found a federated imageboard using ActivityPub, and a library to interact with ActivityPub, so am considering giving this a go once my new machine is setup.

>>27043
Why do you need the LLM? Why not just a basic firewall for now? Also this site sounds pretty sick good luck in your endeavours.

>>27062
Thank you! Avoiding the LLM is probably more practical because of the 2Gb of ram it eats up raising costs 50%. The idea was just to make moderation a little easier for those of us without thick skin.

>>27059
Haven't yet found a way to write an ActivityPub server that would be reasonable. The FChannel implementation is many thousands of lines. The way the've implemented actors is also somewhat strange. Rather than having actors be a salt hashed address or a user account they've made them boards: https://github.com/FChannel0/FChannel-Server/issues/9#issuecomment-822675142 There is a good JSON-LD library from the looks of it.

Have been working on another project (a torrent client which just now started to be able to download files!), but have been growing my todo list for this project. There are a number of related enhancements, CIDR bans which perhaps won't be included (anyone know if these are actually used in practice?), RSS feeds, and one major desired feature addition: ActivityPub.

* Devops
- git repository
- host source
- public instance

* Missing
- caching. (observer pattern?)
- CIDR bans.

these don't make sense with the current interface, and it's not
clear to me whether or not they are required.

- RSS feeds.
- move thread.
- post search.
- ActivityPub.
- OOP restructuring!
- views for ban reports.

requires adding func.now() to these models for ordering.

- incremental autorefresh. (observer pattern?)

this requires not only add and remove but update messages in order
that backlinks be displayed correctly.

* Separate Projects
- booru site.
- LLM moderation.

>>27089
Actually the hold up at the moment is picking a name so can make a new git repo and get to work on the OOP restructure. Was thinking something related to technomancy and like the original "channel" designation. Thought maybe Ars Via, but it's a little much isn't it. Anyone got any ideas?

>>27090
leanboard, webchannel, industrialchan, blackboard, anonpress

>>27091
Ended up rejecting your idea and making the repository just with my previous attempt: Ars Via. Have completed most all of the OOP restructuring. Also used Flask-SQLAlchemy to properly handle the SQL Session. Currently trying to figure out how to go about the incremental autorefresh and caching. Recently learned that for websocket based applications like this one you're really supposed to be using tornado, which is a bit of a bummer.

>>27099
Added caching but am not presently satisfied by my project. To do it right at this point seems to necessitate switching to some sort of asynchronous framework (like Tornado, FastAPI, or Starlette), and an asynchronous ORM to match. Starlettte in particular would probably let me write my socket endpoints using the observer pattern, which is desirable to me. Might not be too difficult to rewrite in this way.

>>27104
>To do it right at this point seems to necessitate switching to some sort of asynchronous framework, and an asynchronous ORM to match.
This has been a little frustrating but the Japanese fusion jazz has helped. It's going to take a little while .. which is good! Spent all day and haven't even managed to get the GET routes up and running.

>>27106
Managed to get a few routes, views, and the render working; migrated to postgres from sqlite, and had to fix up the models a bit to suite. Am apparently going to need to rework the routes, and the asyncio SQLAlchemy extension lacks support for events so there's a bit to do there, and a ton of the views haven't been moved over yet, nor the websockets. Temporarily set posts to work with UUIDs, though something like this would be needed for federation.

>>27128
Was able to add the error pages pretty easily, but session management is giving me some real trouble even with the starlette-login module. Feel the need to take a break from this program; may or may not.

>>27130
Seems like the key is going to be to not use the starlette-login module but rather use the authentication module that comes pre-included.

>>27134
Don't suppose anyone here knows how to make SQLAlchemy return only the first image for each thread? So far as can tell am doing everything correctly here, but still getting an error that "row_number" doesn't exist. If it were just SQL could pull it off but for some reason the "label" even as it emits an AS does not cooperate with the following use of the labeled item.

Unrelated am also having an issue with Starlette-Login trying to serialize user to JSON for some reason and failing at this. Should be able to take a deeper look into this myself at some point.

>>27150
Well one error is that it should be "row_number = 1" rather than "==", but this isn't the problem.

>>27150
>>27151
This issue was solved with the help of another forum.

LOTS of bugs to fix but the rough in of all the functionality sans-websockets for the Starlette/Uvicorn migration is now complete. All the GET routes and views are working, thread posting is working, and so is login. The main question at present is how to setup the GUIDs for the posts. F-Channel uses a random base-32 number of length eight (a namespace of roughly one trillion, if it were base64url it would be two hundred fifty trillion). Another idea is to just use UUID4 (which is 2^128 or 10^38). None of these are really all that appealing to me, but surely will decide on something eventually.

>>27167
Decided on the UUID4 solution. Managed to get posting within a thread, post parsing, bump-ordering, backlinks, and the javascript up and running with the new system. To my knowledge automatic locking of full posts, initialization of admin accounts, GET pages for address and user history, and POST handling for admin, ban, and report still need to be addressed. Hopefully will be able to get through some of these tomorrow.

>>27168
Is there someway to have two hashes for the same object and then confirm that they are for the same object without knowing what that object is? Was thinking for moderation purposes it would be interesting to have hashes of addresses communicated between federated servers along with their respective messages in such a way that each could check whether or not its an already banned user. Found this: https://crypto.stackexchange.com/a/102963 but it seems a little theoretical, so probably not.

>>27169
> automatic locking of full posts, initialization of [default] admin accounts, GET pages for address and user history, and POST handling for admin, ban, and report still need to be addressed.
Believe have solved these now. Hopefully sometime tomorrow can finish off the migrations with its comprehensive testing, and begin to implement new features again.

>>27170
>Hopefully sometime tomorrow can finish off the migrations with its comprehensive testing,
Wouldn't say the testing was quite comprehensive, but did a quick read through, and some sporadic testing which turned out well. Went ahead and merged the Starlette branch with main.

>>27170
>begin to implement new features again.
Am considering getting rid of the (you)s to enable caching to work more effectively. Theoretically post identifiers could be stored in the session, and this way the (you)s added back at a later date. Am also considering replacing the server side rendering of the [logout] button with client side rendering. The downside is slightly less functionality for noscript users, the advantage is that caching is much easier.

>>27171
>Am also considering replacing the server side rendering of the [logout] button with client side rendering.
Some may be happy that this part didn't work out. So will just make template rendering a little more intricate.

>>27171
>>27172
>So will just make template rendering a little more intricate.
This didn't work out either so ended up with a nice and light-weight solution to the problem. Now have caching of all the render functions that query the database. Managed to re-implement the old nonincremental socket-based autorefresh more elegantly in Starlette using the observer pattern that was trying to do with Flask-Sock. Also added a count for how many times the page has been refreshed by the socket. Am pretty happy with all this, though it's not even close to the complexity required to have incremental socket updates. It might be between compression, client-side caching, and server-side caching it wouldn't really be worth the complexity cost for now.

>>27174
Made the image and reply count appear on the catalog page. Also added the user and address reports to the reported page. Been working through bugs otherwise. Here are my remaining TODOs:

Devops
- host source
- public instance

Missing
- RSS feeds.
- ActivityPub.
- post search?

Most of what the program needs at the moment is testing, to uncover bugs and then subsequently fix them. Have been doing some of this already today, but it may be about time to finally get the source code hosted somewhere and start receiving feedback. Biggest hold up here is deciding on the platform; seems like it's GitHub or Codeberg.

>>27174
The likelihood of me actually implementing any of these but hosting the source is quite low come to think of it.

>>27178
Don't have it in me to write documentation at the moment but here's the repository: https://codeberg.org/jung/arsvia Plan to add install instructions, and some pictures. The gist is:
1. install postgres, imagemagick (libmagickwand-dev), python3, and pip.
2. run pip install -e .
3. setup a user and database in postgress and add to config.py
4. python3 arsvia
5. navigate to 127.0.0.1:5000
6. create an account which will automatically become an admin.


>>27176
activitypub is a bad match for imageboards unless you're going to treat every poster as one big account. Activitypub and other protocols seem to assume named accounts which imageboards don't have.

File: 1739165372796.jpeg (7.87 KB, 200x200, xhsdoge.jpeg)

>>28343
Make everyone register an account

(for real tho this could be not bad. It would help against spam and illegal content by raising the barrier to entry some [and new account creation could be temporarily halted, which is better than harming vpn or tor users or etc], and on the front everyone could still be anonymous. Or a tripfag at their choosing. whatever. Would it be so bad?)

>>28343
wait nevermind, the names would still be exposed in the federation process, i'm dumb. I guess every topic/board could be considered a user? I agree though. Also activitypub is just kind of bad in general. It's really nonspecific and everyone implements it so differently that the dream of cross-platform connection is very rarely reality.


Unique IPs: 6

[Return][Go to top] [Catalog] | [Home][Post a Reply]
Delete Post [ ]
[ home / rules / faq ] [ overboard / sfw / alt ] [ leftypol / edu / labor / siberia / lgbt / latam / hobby / tech / games / anime / music / draw / AKM ] [ meta ] [ wiki / shop / tv / tiktok / twitter / patreon ] [ GET / ref / marx / booru ]