archive.wakarimasen.moe is an archiving tech stack designed to explore more efficient and/or cheaper methods to run archives while having a straightforward and east upgrade path. This instance serves as sort of a stress designed to see how things can scale. This project is informally associated with Bibliotheca Anonoma.
The current software stack that powers this board is as follows and is open source.
- Torako - This is a scraper which is designed to be compatible with Asagi. It goes a bit further in providing some options that optimize database load, as well as uploading images to cloud endpoints and pushing posts to a postgres search index. (Configuration)
- FoolFuuka - This boards runs a fork of the FoolFuuka software; changes were made to the Search module to have it support Postgres rather than Sphinx. This will eventually be replaced by Ayase
- MySQL - All posts are served from MySQL 8, using an Asagi compatible schema,
- Postgresql - A separate posts index was created taking advantage of Postgres' Full Text search capabilities.
- Backblaze - Media is stored in Backblaze for cheap storage
- Cloudflare - Cloudflare is used to cheaply serve content
Currently this board is hosted with a cloud provider
Dedicated Machine: Torako
- CPU: 16 CPU
- RAM: 256 GB
- Cost: $65/mo
- Services: Torako, FoolFuuka, MySQL, Postgresql, pgbouncer, nginx, php-7.4
The remaining costs are to Backblaze, which is primarily storage @ $5/TB/mo
This archive fully went live on September 13, 2020 and should have everything from that date. Past posts are going to be imported into the database to study how data size affects performance, depending on how fast that data can be loaded and what data I can get my hands on.
Selected archives my be imported into my own dataset. The following table describes the completed imports.
- /a/ (No Images; There may be some missing images as reposted images that were part of the old dump aren't redownloaded)
DCMA / Abuse
See this page
- Email: admin[@]wakarimasen.moe
- I can be found in the Bibanon Discord