Posts

@Admin
Oof! It’s almost as if ai generated content on Derpibooru wasn’t the real problem and their website should automatically reject too many uploads from the same address/session/user in a short period of time; but what do I know, I’m just another chump with a graphics card.

@Background Pony #0B6C
One guy posting ~20 assorted images of different characters and poses from one artist throughout 3-4 hours pales in comparison to the bimonthly Kevinsano hotdrop on derpi…

Clearly not working already.

@Richard4Ponies
No daily limit, but avoid “spamming” (eg. dumping 20 alts of the same concept in an hour; spread things out).

Quality is much more relaxed than derpi’s “ai composition” tag, but errors (ghost signatures, messy hands, limbs a bit too long, etc.) should still be tagged for those that filter them, and images with really bad errors (like extra limbs) might be removed.

I realized i never asked, but is there still a limit on how many uploads you can do a day like on derpi? Or now just the quality standards?

@Zerowinger
The nightly dump, which contains all the information needed to import, would have been downloaded already, so there is no need to worry.

@Admin
Ah, okay, because Derpi has already purged a large number of AI generated images (all but a handful of mine are gone), and would hope that the comments could at least be salvaged.

@Admin
Maybe split the tag in minor generation errors and major generation errors? Minor gets a pass while major is hidden by default?

This would be an example of minor errors. Window frame is not complete, slightly weird angle and headboard directly at window.
For major errors we have a few examples in my first post.

To use a tag like “low quality” low quality need a proper definition for AI. If I draw a quick doodle of stick figure ponies it would be “low quality” for a human. Most models have a very high “quality” by default if properly utilized.

These are 3 examples were in my opinion a “low quality” tag would be appropriate. The minimalist style was obviously not intentional and the weird body proportions speak of someone not using the base prompts (score_9, score_8_up, …).

Here are some examples what I would consider low detail, but not low quality:


The first is stylized and the second tried to be show accurate.

I know this could be weaponized. Any tag can really. But if we not use anything that can be misused, we would sit still in caves eating meals raw.

Welcome to literally how 99% of people use downvotes as quality is subjective.

@Thoryn
@truekry
Hidden seems a bit harsh, some errors are fairly minor and don’t detract much from the overall image. Spoiler on default would make more sense to me.

Of course, would need a proper spoiler image… maybe the olde cyclops?

(or Derpy… but everyone likes Derpy)

“Low quality” (and others) I feel might be too subjective, though, and vulnerable to weaponization.

@Thoryn
Agree

I still think it would make sense for the default filter to hide images containing tags like generation errors, low quality, machine learning abomination and wtf.

It’s a shame people are using downvotes for things they don’t like, making votes basically useless to determine if something is well done.

@Exuno
(still not a tech person disclaimer)
We downloaded last night’s public derpi dump, plan is to import the comments from that this weekend.

The deletion date in derpi seems to have passed; and I’m only seeing the comment counts here, not the actual content. Is it still importable, or was there some kind of miscommunique?

Thanks for the update. Glad something like that was caught quickly.

@Admin
For those wondering what this is about, and keeping in mind I’m not a tech person…

Basically to speed things up (and in case the AI deletions happened before the import did) derpi gave us access to a comment dump. The dump included all comments, which without realizing included the names of as-anon commenters; the import script could not have possibly caught this error, since it was given commenter names as plain text without the as-anon flag, so names were used instead of anon ids. As soon as that was reported to us the import was reverted.

We’ll be running another import soon using the standard public dump instead (with a side effect of also pushing back the deletion date on derpi).

AFAIK derpi’s tech admin will be making a post about it there as well with I assume better details.

Comments temporarily reverted due to an issue we encountered, will be reimported Soon. Apologies for the inconvenience.

Thanks, I know too well about having things on a to-do list and never being able to get to them due to other priorities.

Funnily enough, the script would have been way simpler with direct access to both the Derpi and Tantabus databases and just having it directly import instead of having to process the nightly dump.

But just in case direct connection to both wasn’t possible, I went with that method, especially if the data was already deleted on Derpi.

Anyways, I have a lot of experience with these types of things, especially data migration between services, and I knew I could get something working fairly quickly.

Anyways, if there is anything else that needs migrating, let me know.

I’m sure this community would prefer to have the tags with the correct details, such as categories, aliases, includes, and descriptions, etc.

I’d like to personally thank you, @Adorable Blue Fox for the incredible script.

If it wasn’t for your effort, I am pretty sure “migrate the comments” would have remained a task somewhere on my TODO list, potentially indefinitely as it wasn’t that high of a priority.

The code was easy to understand, and dealt with all the potential issues such migration could easily encounter (Most notably, re-runs, without risking duplications)

Really, thank you! :)

@Adorable Blue Fox
There’s still a ludicrous amount of stuff being tagged ai composition over there that’s going to need heavy-handed enforcement or this whole thing is pointless. I can’t speak to what is actually being enforced but ‘ai that looks good enough’ or ‘ai with a speech bubble’ shouldn’t be the standard for what is acceptable.

@Adorable Blue Fox
when all of these iterations of programmatic transfers are done we’re going to be going through any images with sus tag changes by hand to make sure we aren’t missing any that should be here

Ah, okay, that makes sense.

There are people who retagged images from ai generated to ai composition, etc. Still, if the plan is to import the rest “soon,” then it would at least just cover that base even though they’re both going to be on Derpi and Tantabus. Who knows, the rules could change again.

@Adorable Blue Fox
We run it only for the “ai generated” images, as (the majority of) those will be removed from derpi. We’ll grab the rest “soon”, though we likely won’t be rerunning it for updates after that.

Woo, thanks for running my script; I am glad to see the comments on the tantabus now.

Also, are the comments still being imported? Because I noticed a few missing comments, which I never had issues with during my tests. If someone could send me the processing.log file, that would be great so I can fix that potential issue.

Here is a small example that does not have the comments imported:
Tantabus & Derpi
And to sanity check that comment does exist on my test environment:

I’ve also noticed that I neglected to check for deleted comments to prevent them from being imported, so any deleted/banned comments are now visible again as raw text. I can adjust the script to fix that issue.

@Background Pony #7570
Yes!

We were planning on making a proper post about it (combined with a couple other things), but yep, everything worked out there.

Huge thanks to Adorable Blue Fox, our sysadmin was very impressed with the functionality and quality!