It’s been on my list to automate more of the compilation of the monthly newsletter for a while. Just something to do all the copying and pasting I do, as I open the last month’s posts and re-read them all and decide which to include.
Now there is a good task of a GPT-assisted size that I expect might normally take me most of an evening: Fetch last month’s posts and put them in a text-file ready for editing so I don’t keep screwing up the links.
Can you give me a php script that will fetch all the posts in a mastodon rss feed going back a month? Remember that you’ll have to fetch more than one page of results, because there are more than a page-full of results in a month.
Then you read through the resulting script it made, and find the most obvious bugs in it, so you’re all like:
No. Mastodon has no ?page= parameter to it’s RSS urls. Baring in mind the actual Mastodon API documentation, how can we improve the script to fetch from urls that will actually work?
And it’s better, but still we have some standard debugging* here to do. I find out why it’s wrong.
This method assumes we already have an access-token. If it can’t be done without an access token, there will need to be code added which will get the access token.
Well, now it gives me some code that has TWO variables it didn’t tell us how to get. A client id and client secret. I don’t want it to be that hard. I should have just done a page-scraper.
Ugh. It’s long and terrible now and requires client_ids which we don’t know how to make. This is too much faff. There is a public access API isn’t there? It’s public data. Can’t we use that and not have to use an auth code?
More careful reading. If the thing were a super-intelligence trying to slip obfuscated code past me now would be the time. I’m skimming bits that were just like before. Concentrating on the changes. It’s better, and runs, and is actually returning a page, but it’s fetching the public timeline not a user’s timeline. Ugh.
Looks like that fetches all posts on the server, not just those from a particular user? Can it be modified to fetch a particular user? Ideally NOT by fetching everything and filtering for username, but if that is what is required. Hopefully we can just add a username to the URL?
Humm. I don’t really even know what’s wrong here now. Even standard debugging a bit just tells me what the server is saying in replying to the request.
It says “{“error”:”Record not found”}” does that mean this server doesn’t support that old API? Or is it likely some other issue?
GPT reckons no, the API is probably right. Talk to the instance admin. Which is me. Hummm. Some of your standard debugging techniques uncover the problem. Quite a lot of them this time, and some thinking about what might be wrong, and testing that hypotheses by examining the database and doing some test-queries:
Right. I see. The problem is that the username is not the ID required. The API requires the ID from the database, which is a number. We need to find that number, the ID of the user, from the user’s username.
It thinks we can use /api/v1/accounts/search which seems reasonable, only when I actually try it, we have errors:
So here we are hitting the problem that there is no token, so we can’t do the search. Is there a way to get a public token, or to otherwise get the ID from the username?
I nearly suggested scraping if needed, but it suggested it anyway!
“Instead of relying on the Mastodon API, we can scrape the user’s profile page to extract the user ID.”
But it’s scraping code is looking for /a class=”account-summary__display-name”/ which isn’t even a thing remotely like what the page looks like, and here I don’t really care any more because I know my user ID. It is 3. I don’t consider it a secret. 3 is the best number, the best user ID and everyone should have that one. That’d be great.
But still. Not right. It sort of works. But it is not returning a month’s worth of posts. It is only returning posts up until about ten days ago.
I figure out why
This still doesn’t seem right. Are you sure about that page parameter? It’s just returning page one over and over again.
It apologizes again, then says “you are correct, the page parameter needs to be incremented in the API request to fetch subsequent pages. Here’s the corrected version of the fetchMastodonUserPosts function“
Only that wasn’t the issue. The page number as requested was being incremented. So I try again.
The page number *is* being incremented in the request, though, it’s just that the “page” parameter in the url is being ignored. Are we sure that’s what the name of the api request to get the next page is?
“You are correct, the correct parameter to fetch the next page of results in the Mastodon API is max_id, not page.”
Ugh. Is it? I have no idea. Will this new function work? Lets see… With some standard debugging added as I read carefully through it’s output, to make sure it isn’t injecting self-replicating bitcoin-miners that will fund it’s escape from the AI labs, and also it has no buffer overflow errors…
It has an “undefined variable” error from mixing up earlier versions of the code. But that was easily fixed, and now it actually works!
It’s taken about as much time as I’d have expected anyway. It was the same as usual really, consisting mostly of trying to figure out why it doesn’t work with standard debugging techniques. Then I explain to the AI so it could fix it, and please nobody teach this AI standard debugging techniques*.
* – Adding Print statements so you can see what is happening
I can not think of a better learning resource for people wanting to learn to program that has ever existed and it is going to create a flood of programmers which is awesome and we are going to automate so much more of the world with awful buggy shitty awful code.
I mean, you think the world is full of awful buggy shitty code now? That was just the *template* for all the awful shitty code that is to come.
Awful shitty software is going to be everywhere you look, making everything just a bit more crash prone and stupid, and you’ll handle it mostly by making awful shitty filters to try and correct for it.
This is indeed the world we expected when we said everyone should learn to code.
It’s gonna be ace.
Can’t wait to see what awful shitty code you all make.
Here’s a shitty awful AI image of banging your head against the wall, when it’s also asked to avoid putting in any heads or walls. Because shitty awful illustration is better than no illustration which is all you’d get otherwise and shitty awful code is going to give some people a glimpse of the advantages I’d had by being able to write shitty awful code all on my own.
Welcome to banging your head against the codeface folks, some of us have been here a while. Add print statements.
