Cool! Our park has disappeared again this morning! 😱 Also it was cold outside! 🥶
5 hours ago
💬 Reply
abucci https://anthony.buc.ci/user/abucci/twtxt.txt Remove
mckinley https://twtxt.net/user/mckinley/twtxt.txt Remove
mckinley@mckinley.cc https://mckinley.cc/twtxt.txt Remove
ocdtrekkie https://twtxt.net/user/ocdtrekkie/twtxt.txt Remove
off_grid_living https://twtxt.net/user/off_grid_living/twtxt.txt Remove
prologic
Cool! Our park has disappeared again this morning! 😱 Also it was cold outside! 🥶
5 hours ago
💬 Reply
prologic
Reply to #bbnfuvq
@lyse I'll fix it tonight Sadly I have to rebuild the index 🤦♂️
5 hours ago
💬 Reply
prologic
Reply to #bgoecxa
@lyse This ☝️
5 hours ago
💬 Reply
abucci
Reply to #2rxkcca
@prologic I don't think it's your code. As you said in one of your commit comments, the internet is a hostile place! That's partly why I reacted the way I did: all things considered it's usually better to react quickly and clean up the mess later, then it is to wait and risk further damage. Anyway it sucks @xuu got caught up in it. Hopefully it's all good now.
13 hours ago
💬 Reply
prologic
Oh I forgot again 🤦♂️ Last Saturday of the month, so if anyone's up for a friendly catch up over video tomorrow? Same time, same place 👌
14 hours ago
💬 Reply
prologic
Reply to #kwepmhq
@bender Weird dunno what to say🤣
14 hours ago
💬 Reply
prologic
Reply to #kwepmhq
@bender Huh? 🤔
15 hours ago
💬 Reply
prologic
Reply to #2rxkcca
Also FWIW this is all my fault for writing shitty vulnerable code 🤣 So blame me! I'm sorry 🙏
23 hours ago
💬 Reply
prologic
Reply to #2rxkcca
FWIW I'm still trying to find the the cause of the mult-GB avatars that both @stigatle and @abucci 's pods were both teying yo download. The flaw has since been fixed in the code but I'm still trying to investigate the source 🤞
23 hours ago
💬 Reply
abucci
Reply to #2rxkcca
@xuu I hope everything is sorted out with your ISP. Please let me know if there's anything I can do to help. I sincerely did not mean to cause you any trouble.
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@stigatle @xuu @lyse "Not cool"? I was receiving many broken (HTTP 400 error) requests per second from an IP address I didn't recognize, right after having my VPS crash because the hard drive filled up with bogus data. None of this had happened on this VPS before, so it was a new problem that I didn't understand and I took immediate action to get it under control. Of course I reported the IP address to its abuse email. That's a 100% normal, natural, and "cool" thing to do in such a situation. At the time I had no idea it was @xuu .
The moment I realized it was @xuu and definitely a false alarm, I emailed the ISP and told them this was a false positive and to not ban or block the IP in question because it was not abusive traffic. They haven't yet responded but I do hope they've stopped taking action, and if there's anything else I can do to certify to them that this is not abuse then I will do that.
I run numerous services on that VPS that I rely on, and I spent most of my day today cleaning up the mess all this has caused. I get that this caused @xuu a lot of stress and I'm sincerely sorry about that and am doing what I can to rectify the situation. But calling me "not cool" isn't necessary. This was an unfortunate situation that we're trying to make right and there's no need for criticizing anyone.
1 day ago
💬 Reply
prologic
Reply to #2qczosq
@bender Hehe 🤣
1 day ago
💬 Reply
prologic
Hmmm something happened last night at ~3am (AEST) that decrased traffic to my pod quite considerably... Hmmm? Anyone have any ideas? 💡 
1 day ago
💬 Reply
prologic
Reply to #rdeh4dq
@abucci No worries! All in the name of better reliability and security 😅
1 day ago
💬 Reply
prologic
Reply to #4ljpfuq
@stigatle Thanks! Sooo cold 🥶
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@prologic Yes, I can do that.
1 day ago
💬 Reply
prologic
Reply to #2rxkcca
@stigatle no problems 👌 one problem solved at least 🤣
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@stigatle @prologic my `/tmp` is also fine now! Thanks for your help @prologic!
1 day ago
💬 Reply
prologic
Anyway, I'm gonna have to go to bed... We'll continue this on the weekend. Still trying to hunt down some kind of suspected mult-GB avatar using @stigatle 's pod's cache:
```
$ (echo "URL Bytes"; sort -n -k 2 -r < avatars.txt | head) | column -t
URL Bytes
https://birkbak.neocities.org/avatar.jpg 667640
https://darch.neocities.org/avatar.png 652960
http://darch.dk/avatar.png 603210
https://social.naln1.ca/media/0c4f65a4be32ff3caf54efb60166a8c965cc6ac7c30a0efd1e51c307b087f47b.png 327947
...
```
But so far nothing much... Still running the search...
1 day ago
💬 Reply
prologic
Reply to #2rxkcca
Out of interest, are you able to block whole ASN(s)? I blocked the entirely of teh AWS and Facebook ASN(s) recently.
1 day ago
💬 Reply
prologic
Reply to #2rxkcca
@abucci Oh 🤣 Well my IP is a known subnet and static, so if you need to know what it is, Email me 😅
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@prologic I unbanned a few IP address I had blocked before the bugfix. I wasn't being careful and just blocked any IP I saw making a large number of requests to my pod. That slowed the problem down but I think I blocked your and @stigatle 's pods in the process, oops.
1 day ago
💬 Reply
prologic
Reply to #2rxkcca
@abucci Seems to be okay now hmmm
1 day ago
💬 Reply
prologic
Reply to #qv5sgja
@abucci Hmm I can see your twts on my pod now 🤔
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@stigatle Sweet, thank you! I've been shooting myself in the foot over here and want to make sure the situation is getting fixed!
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@stigatle @prologic testing 1 2 3 can either of you see this?
1 day ago
💬 Reply
abucci
Hmm, I wonder if I banned too many IPs and caused these issues for myself 😆
1 day ago
💬 Reply
abucci
twts are taking a very long time to post from `yarn` after the latest upgrade. Like a good 60 seconds.
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@prologic I don't know if this is new, but I'm seeing:
```
Jul 25 16:01:17 buc yarnd[1921547]: time="2024-07-25T16:01:17Z" level=error msg="https://yarn.stigatle.no/user/stigatle/twtxt.txt: client.Do fail: Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)" error="Get \"https://yarn.stigatle.no/user/stigatle/twtxt.txt\": dial tcp 185.97.32.18:443: i/o timeout (Client.Timeout exceeded while awaiting headers)"
```
I no longer see twts from @stigatle at all.
1 day ago
💬 Reply
abucci
Reply to #2rxkcca
@prologic Have you been seeing any of my replies?
1 day ago
💬 Reply
prologic
@abucci / @abucci Any interesting errors pop up in the server logs since the the flaw got fixed (_unbounded `receieveFile()`_)? 🤔
1 day ago
💬 Reply
prologic
Hmmm 🧐
```
for url in $(jq -r '.Twters[].avatar' cache.json | sed '/^$/d' | grep -v -E '(twtxt.net|anthony.buc.ci|yarn.stigatle.no|yarn.mills.io)' | sort -u); do echo "$url $(curl -I -s -o /dev/null -w '%header{content-length}' "$url")"; done
...
```
😅 Let's see... 🤔
1 day ago
💬 Reply
abucci
Reply to #wpadxra
It shows up in my twtxt feed so that's good.
1 day ago
💬 Reply
abucci
This is a test. I am not seeing twts from @stigatle and it seems like @prologic might not be seeing twts from me. Do people see this?
1 day ago
💬 Reply
abucci
@prologic I am not seeing twts from @stigatle anymore. Are you seeing twts from me?
1 day ago
💬 Reply
prologic
Reply to #ve43paq
@stigatle The one you sent is fine. I'm inspecting it now. I'm just saying, do yourself a favor and nuke your pod's garbage cache 🤣 It'll rebuild automatically in a much more prestine state.
1 day ago
💬 Reply
prologic
Reply to #ve43paq
That was also a source of abuse that also got plugged (_being able to fill up the cache with garbage data_)
1 day ago
💬 Reply
prologic
Reply to #ve43paq
Ooof
```
$ jq '.Feeds | keys[]' cache.json | wc -l
4402
```
If you both don't mind dropping your caches. I would recommend it. Settings -> Poderator Settings -> Refresh cache.
1 day ago
💬 Reply
abucci
Reply to #ve43paq
@prologic
```
./tools/dump_cache.sh: line 8: bat: command not found
No Token Provided
```
I don't have `bat` on my VPS and there is no package for installing it. Is `cat` a reasonable alternate?
1 day ago
💬 Reply
prologic
Reply to #ve43paq
@stigatle Thank you! 🙏
1 day ago
💬 Reply
abucci
Reply to #ve43paq
@prologic Try hitting this URL:
https://twtxt.net/external?nick=nosuchuser&uri=https://foo.com
Change `nosuchuser` to any phrase at all.
If you hit https://twtxt.net/external?nick=nosuchuser , you're given an error. If you hit that URL above with the `uri` parameter, you can a legitimate-looking page. I think that is a bug.
1 day ago
💬 Reply
prologic
Reply to #ve43paq
@stigatle Ta. I hope my theory is right 😅
1 day ago
💬 Reply
abucci
Reply to #ve43paq
@prologic Hitting that URL returns a bunch of HTML even though there is no user named `lovetocode999` on my pod. I think it should 404, and maybe with a delay, to discourage whatever this abuse is. Basically this can be used to DDoS a pod by forcing it to generate a hunch of HTML just by doing a bogus GET like this.
1 day ago
💬 Reply
prologic
Reply to #ve43paq
But just have a look at the `yarnd` server logs too. Any new interesting errors? 🤔 No more multi-GB tmp files? 🤔
1 day ago
💬 Reply
prologic
Reply to #ve43paq
@stigatle You want to run `backup_db.sh` and `dump_cache.sh` They pipe JSON to stdout and prompt for your admin password. Example:
```
URL=<your_pod_url> ADMIN=<your_admin_user> ./tools/dump_cache.sh > cache.json
```
1 day ago
💬 Reply
abucci
Reply to #ve43paq
I'm seeing GETs like this over and over again:
```
"GET /external?nick=lovetocode999&uri=https://vuf.minagricultura.gov.co/Lists/Informacin%20Servicios%20Web/DispForm.aspx?ID=8375144 HTTP/1.1" 200 35861 17.077914ms
```
always to `nick=lovetocode999`, but with different `uri`s. What are these calls?
1 day ago
💬 Reply
prologic
Reply to #ve43paq
Just thinking out loud here... With that PR merged (_or if you built off that branch_), you _might_ hopefully see new errors popup and we might catch this problematic bad feed in the act? Hmmm 🧐
1 day ago
💬 Reply
prologic
Reply to #ybzi67q
@slashdot I _thought_ Sunday was the hottest day on Earth 🤦♂️ wtf is wrong with Slashdot these days?! 🤣
1 day ago
💬 Reply
prologic
Reply to #ve43paq
if we can figure out wtf is going on here and my theory is right, we can blacklist that feed, hell even add it to the codebase as an "asshole".
1 day ago
💬 Reply
prologic
Reply to #ve43paq
@stigatle The problem is it'll only cause the attack to stop and error out. It won't stop your pod from trying to do this over and over again. That's why I need some help inspecting both your pods for "bad feeds".
1 day ago
💬 Reply
