Opening Thread

@Background Pony #7570
Yes!

We were planning on making a proper post about it (combined with a couple other things), but yep, everything worked out there.

Huge thanks to Adorable Blue Fox, our sysadmin was very impressed with the functionality and quality!

Woo, thanks for running my script; I am glad to see the comments on the tantabus now.

Also, are the comments still being imported? Because I noticed a few missing comments, which I never had issues with during my tests. If someone could send me the processing.log file, that would be great so I can fix that potential issue.

Here is a small example that does not have the comments imported:
Tantabus & Derpi
And to sanity check that comment does exist on my test environment:

I’ve also noticed that I neglected to check for deleted comments to prevent them from being imported, so any deleted/banned comments are now visible again as raw text. I can adjust the script to fix that issue.

@Adorable Blue Fox
We run it only for the “ai generated” images, as (the majority of) those will be removed from derpi. We’ll grab the rest “soon”, though we likely won’t be rerunning it for updates after that.

Ah, okay, that makes sense.

There are people who retagged images from ai generated to ai composition, etc. Still, if the plan is to import the rest “soon,” then it would at least just cover that base even though they’re both going to be on Derpi and Tantabus. Who knows, the rules could change again.

@Adorable Blue Fox
when all of these iterations of programmatic transfers are done we’re going to be going through any images with sus tag changes by hand to make sure we aren’t missing any that should be here

@Adorable Blue Fox
There’s still a ludicrous amount of stuff being tagged ai composition over there that’s going to need heavy-handed enforcement or this whole thing is pointless. I can’t speak to what is actually being enforced but ‘ai that looks good enough’ or ‘ai with a speech bubble’ shouldn’t be the standard for what is acceptable.

I’d like to personally thank you, @Adorable Blue Fox for the incredible script.

If it wasn’t for your effort, I am pretty sure “migrate the comments” would have remained a task somewhere on my TODO list, potentially indefinitely as it wasn’t that high of a priority.

The code was easy to understand, and dealt with all the potential issues such migration could easily encounter (Most notably, re-runs, without risking duplications)

Really, thank you! :)

Thanks, I know too well about having things on a to-do list and never being able to get to them due to other priorities.

Funnily enough, the script would have been way simpler with direct access to both the Derpi and Tantabus databases and just having it directly import instead of having to process the nightly dump.

But just in case direct connection to both wasn’t possible, I went with that method, especially if the data was already deleted on Derpi.

Anyways, I have a lot of experience with these types of things, especially data migration between services, and I knew I could get something working fairly quickly.

Anyways, if there is anything else that needs migrating, let me know.

I’m sure this community would prefer to have the tags with the correct details, such as categories, aliases, includes, and descriptions, etc.

Comments temporarily reverted due to an issue we encountered, will be reimported Soon. Apologies for the inconvenience.

@Admin
For those wondering what this is about, and keeping in mind I’m not a tech person…

Basically to speed things up (and in case the AI deletions happened before the import did) derpi gave us access to a comment dump. The dump included all comments, which without realizing included the names of as-anon commenters; the import script could not have possibly caught this error, since it was given commenter names as plain text without the as-anon flag, so names were used instead of anon ids. As soon as that was reported to us the import was reverted.

We’ll be running another import soon using the standard public dump instead (with a side effect of also pushing back the deletion date on derpi).

AFAIK derpi’s tech admin will be making a post about it there as well with I assume better details.

Thanks for the update. Glad something like that was caught quickly.

The deletion date in derpi seems to have passed; and I’m only seeing the comment counts here, not the actual content. Is it still importable, or was there some kind of miscommunique?

@Exuno
(still not a tech person disclaimer)
We downloaded last night’s public derpi dump, plan is to import the comments from that this weekend.

@Admin
Ah, okay, because Derpi has already purged a large number of AI generated images (all but a handful of mine are gone), and would hope that the comments could at least be salvaged.

@Zerowinger
The nightly dump, which contains all the information needed to import, would have been downloaded already, so there is no need to worry.

I realized i never asked, but is there still a limit on how many uploads you can do a day like on derpi? Or now just the quality standards?

@Richard4Ponies
No daily limit, but avoid “spamming” (eg. dumping 20 alts of the same concept in an hour; spread things out).

Quality is much more relaxed than derpi’s “ai composition” tag, but errors (ghost signatures, messy hands, limbs a bit too long, etc.) should still be tagged for those that filter them, and images with really bad errors (like extra limbs) might be removed.

Clearly not working already.

@Background Pony #0B6C
One guy posting ~20 assorted images of different characters and poses from one artist throughout 3-4 hours pales in comparison to the bimonthly Kevinsano hotdrop on derpi…

@Admin
Oof! It’s almost as if ai generated content on Derpibooru wasn’t the real problem and their website should automatically reject too many uploads from the same address/session/user in a short period of time; but what do I know, I’m just another chump with a graphics card.

Honestly, if there’s one thing Derpi/Philomena desperately needs, it’s parent/child posts.