All things need to come to an end.
One of the reasons while I started freelancing in late 2020 was that I had a "bucketlist" of projects I wanted to work on. But then I was lucky enough to get projects right from the beginning and was doing that for a while. November 2022, after a pretty good year, I took some time off to travel back to Japan, the country where my mother is from, that I hadn't visited in 12 years. I came back and chilled most of December. In January I still had no new projects, and it was also the time when all the big companies were laying off people as Covid seemed to finally be over, and it looked like it would stay quiet for a bit.
This felt like exactly the right time to come back to my project list. I started thinking about it, how much work it would be, how much I already knew about how to do them, what I had to learn, and so on. Eventually I settled on what would later become fedistats. The idea was to tap into the stream of content that's flowing through social media to unearth trends and interesting news. I had already done something similar a couple of years back so I had an idea how to get started.
Looking back, I cannot believe that this was just two years ago. Back then LLMs had just made an entry and there was a lot of hype but not that many real tangible benefits. Fedistats was a decidedly pre-AI project. It was just doing statistics on hashtags and links. It didn't use "AI" to understand text, no RAGs, no vector databases. Just a lot of counting and postgres.
This was also way before coding assistants. I was literally typing each single byte into a keyboard to create the whole thing!
And it was a fun project! The original premise definitely worked out, at least for high volume topics like AI (ironically). I built an "automated newsletter" that looked at top 10 links that were posted with the #AI hashtag, and that produced a constant stream of interesting news. For other, more niche topics, it wasn't working that well. Also, there is a lot of noise on mastodon. But whenever something big happened, you could definitely see it in the trends.
It also taught me a lot about building a project people actually need. It was way too technical and worked for me as an analytics engine, but the average user was overwhelmed. It also targeted people who were interested in news, but didn't want to follow social media, but all my "marketing" was done on social media...
The project let me explore some of the challenges of dealing with this amount of data on a single smallish virtual machine (I didn't want to spend more money on a side project). Again I made the mistake of using postgres as an analytics engine. I switched to precomputing all statistics and using postgres just for retrieval, filtering, and search. I didn't initially think about how to clean out data eventually again, so that the machine would run out of memory again and again. I had discussions with friends who said "why don't you just buy a bigger machine? That $5 more is so much cheaper than you working on it on the weekends!" And there were right, but it also just postponed dealing with the problem.
A couple of weeks ago fedistats ran out of memory again, and I realized that it was time to say goodbye. I'll keep the website around longer and the topics with all the interesting links it found. You should also still be able to unsubscribe through the link that's at the bottom of each email.
The past few months I started to really get into vibe coding, and that has enabled me to work again at the bucketlist of projects at a level of productivity that was unimaginable before. Stay tuned! And thanks for reading!
You can find the topics here.
If you want to learn more about my journey and the fediverse, follow me on Bluesky or on Mastodon, or follow fedistats on Mastodon.