Elder Scrolls
Advertisement
Elder Scrolls
Main article: User:Atvelonis/Bot

discussions_delete.py scans posts made to the Discussions in a certain timeframe and then deletes ones that are in a blacklist. See Flightmare's original script for the code this was adapted from.

discussions_delete.py relies on another file called core.py in order to log in. They must exist in the same directory (folder) in order to function correctly. See the How-to guide section for installation and operation instructions.

The AutoDMod bot, operated by SOAP, has a similar purpose as the discussions_delete.py script, though it is written in JavaScript and not publicly accessible.

My documentation is written for Windows 10 users. Linux and OSX users may receive different error messages.

API data[]

Main article: User:Atvelonis/Bot/Discussions API

The script itself relies on a set of API calls to perform actions. One set is to the MediaWiki API, for login, and another is to the Discussions API, for interaction on Discussions. A bot password must be set up in order to access the wiki from the MediaWiki API.

Complete file[]

# Credit: Atvelonis, Flightmare
import core
import json
import requests
import time
import re
import pathlib
import sys
"""
core.py: for login
json: https://docs.python.org/3/library/json.html
requests (cmd: 'py -m pip install requests'): http://docs.python-requests.org/en/master/
time: https://docs.python.org/3/library/time.html
re: https://docs.python.org/3/library/re.html
pathlib: https://docs.python.org/3/library/pathlib.html
"""

headers = {'Accept': 'application/hal+json', 'User-Agent': '<Username>/Bot'} # HTTP header
wiki = '<wiki>'
api = 'discussions_api' # discussions_api or mediawiki_api
session = core.login(api)

payload = {'limit': '25', 'page': '0', 'responseGroup': 'small', 'reported': 'false', 'viewableOnly': 'true'}
req = session.get('https://'+wiki+'.fandom.com/wikia.php?controller=DiscussionPost&method=getPosts', params=payload, headers=headers)
#decoded_json = req.json()
print(req) # HTTP status; should be 200

ts = time.time()
update_interval = 60 # seconds

# Views posts made in the last two minutes and deletes them if needed
for post in reversed(req.json()['_embedded']['doc:posts']):
    if int(post['creationDate']['epochSecond']) > ts - update_interval:
        content = post['rawContent'].casefold()
        name = post['createdBy']['name'].casefold()
        avatar = post['createdBy']['avatarUrl']
        forum_name = post['forumName']
        thread_id = post['threadId']
        user_id = post['createdBy']['id']
        post_id = post['id']
        thread_title = post['_embedded']['thread'][0]['title'].casefold()
        post_epoch = post['creationDate']['epochSecond']
        json_model = post['jsonModel'] # to catch links
        
        if post['isReply']:
             thread_title = post['_embedded']['thread'][0]['title'] + ' (reply)'
        is_reply = post['isReply']
        
        #print(name + ', thread: "' + thread_title + '": ' + content + "\n\n----")
        
        # Content the bot deletes. Must be lowercase
        blacklist = ['foo', 'bar', r'foo+bar']
        for i in blacklist:
            if re.search(i, content) or re.search(i, thread_title) or re.search(i, name):
                result = session.post('https://'+wiki+'.fandom.com/wikia.php?controller=DiscussionPost&method=delete&postId='+post_id, headers=headers)
                print(result)
                print('Deleted: https://'+wiki+'.fandom.com/d/p/'+thread_id+'/r/'+post_id+' and content was: '+content)

                # Log each deleted entry to a file for future reference/debugging
                with open(pathlib.Path('C:/Users/<Username>/Desktop/AkulakhanBot_Wiki_Log.txt'), 'at') as logging_file:
                    logging_file.write('Deleted: https://'+wiki+'.fandom.com/d/p/'+thread_id+'/r/'+post_id + \
                                       '\nTimestamp: ' + str(post_epoch) + \
                                       '\nDeleted post title: ' + thread_title + \
                                       '\nDeleted post contents: ' + content + \
                                       '\nUser: ' + name + \
                                       '\nBlacklist: ' + i + '\n\n')
                    logging_file.close()
                print('Logged deleted post to file')

Deletion[]

There are a bunch of ways you can design a blacklist. If you aren't using regular expressions, I'd suggest iterating through a list, as shown above. This code will look at the value of the Discussions post (stored in the variable content) and then go through each term in the list that I've named blacklist, looking for a match.

The for loop obviously has to be followed by the session.put line to do anything. The print statements are optional.

The following is the syntax for a list that includes both normal terms and regular expressions. The latter are signified by an r preceding the string, and any regex syntax must be included within the string.

blacklist = ['foo', 'bar', r'foo+bar']
for i in blacklist:
    if re.search(i, content):

You can also format each of the terms on a new line:

blacklist = ['foo',
             'bar',
             r'foo+bar']
for i in blacklist:
    if re.search(i, content):

You don't actually have to use an array if you just want to blacklist a single term, but this syntax makes it very easy to expand the blacklist, which you will almost definitely want to do down the line.

Output[]

Optional print to display which exact terms were caught by the filter
print(re.findall(r"(?=("+'|'.join(blacklist)+r"))",content))
Optional code to make the bot write a log to a predefined text file

"a" means "append" (i.e. add to an existing file, rather than overwriting), and "t" means "text" (as opposed to "b" for "bytes"). See documentation for the Python "open" function for more information.

logging_file = open(pathlib.Path('C:/Users/<Username>/Desktop/AkulakhanBot_Log.txt'), 'at')
logging_file.write('Deleted: https://'+wiki+'.fandom.com/d/p/'+thread_id+'/r/'+post_id + \
                   '\nTimestamp: ' + str(post_epoch) + \
                   '\nDeleted post title: ' + thread_title + \
                   '\nDeleted post contents: ' + content + \
                   '\nUser: ' + name + \
                   '\nBlacklist: ' + i + '\n\n')
logging_file.close()

You could also do this with pickle, but there's no reason to do so.

logging_file = pathlib.Path('C:/Users/<Username>/Desktop/AkulakhanBot_Log.txt'
blacklist = ['foo', 'bar']
for i in blacklist:
    pickle.dump('Deletion trigger for '+https://'+wiki+'.fandom.com/d/p/'+thread_id+'/r/'+post_id+': '+i+'\n', open(logging_file), 'ab'))

How-to guide[]

If you're reading this, I probably unhelpfully linked to this documentation with the expectation that you would be able to figure it all out on your own, or I have not managed to get our bus factor above one and have paid the price. Either way, running the bot is actually a lot easier than you think it is. It does not really require any understanding of programming, unless you wish to modify an existing file; I have already written the code. The setup will take no more than a few minutes. Here's how you do it.

Set up bot login
  1. Create an empty text file (.txt) in an accessible location on your computer (wherever you want) named "BotNameLogin.txt" (but it can be anything) to store session data. Just make sure that you know the name and the filepath!
  2. Download Python and follow the installation instructions. I would recommend running the newest version.
  3. Open up the Command Prompt, type in py -m pip install requests, and press Enter. Follow any instructions that it gives you. This will install one of the libraries necessary for the bot to function.
  4. Open IDLE (Python's text editor) or another such program. Create a new file, copy+paste the contents of core.py into it, and then save.
    1. In core.py, replace the value of the login_file variable, C:/Users/<Username>/Desktop/Wiki_Bot/AkulakhanBotLogin.txt, with the name of the empty text file that you just created, and its filepath.
    2. If you're not sure what the filepath is, you can right-click on the file and select "Properties." The filepath will be listed under "Location."
    3. If you used any backslashes \, replace these with forward slashes /. Your username obviously does not need to be surrounded by <>; just write it as-is.
  5. Create another empty text file (.txt) n an accessible location on your computer (wherever you want) named "BotName_Login_Credentials.txt" (but it can be anything) to store your bot's username and password.
    1. On the first line, put the bot's username. I highly recommend using a bot account that is separate from your main account.
    2. On the second line, put the bot's normal password.
    3. On the third line, put the bot's API password (the one you get from Special:BotPasswords).
    4. In core.py, replace C:/Users/<Username>/Desktop/Wiki_Bot/AkulakhanBot_Login_Credentials.txt with the filename and filepath you used.
Set up deletion script
  1. In IDLE, create a second new Python file, and copy+paste the contents of discussions_delete.py (or whatever you need) into it. If you are operating a bot on the Discussions to act as an Abuse Filter, for example, you will want the one labeled here as discussions_delete.
  2. Find the variable called wiki and fill them in according to your wiki and account. The wiki name that you choose should be what you enter into the URL.
  3. Under the variable headers (used at least three times), you should technically also change the value of User-Agent from <Username>/Bot to whatever your username is, but this will not affect whether or not the script runs, just how some data is handled.
  4. Replace the content that's pre-written in the blacklist with the content that you want to find+replace. If this is a Discussions bot meant to delete posts with certain keywords, you will also want to use a Python list instead of a thousand or statements. To do this, replace the line if 'Like, C0DA makes it canon, dude.' in content: with blacklist = ['foo', 'bar']<new line>if any(i in content for i in blacklist):. Make sure that the indentation is exactly the same. Then replace foo, bar, etc. with individual keywords that you want to add to the blacklist. The syntax for using regular expressions here is shown above.
  5. If your script is supposed to run on a continual basis, modify the value of update_interval (in seconds) as you see fit. This is how far back the bot will scan in a given feed.
Set up automation
  1. If you're operating a script continually (rather than just once), such as a recurring Discussions spam removal script, open Task Scheduler on Windows.
  2. In the top-right corner, click "Create Task." Name it and give it a description. Click "Run whether the user is logged on or not." Otherwise, you will get a lot of annoying popups every time it runs. Set "Configure for" to "Windows 10" or whatever your operating system is.
  3. Go to the "Triggers" tab and click "New." Check the box that says "Repeat task every 1 hour," and replace "1 hour" with the value that you wrote for update_interval in your Python file. For example, if the parameter of the variable is 60, then in Task Scheduler you will want to say "1 minute." Then press "OK."
  4. Go to the "Actions" tab and click "New." Make sure that you have it set to "Start a program" by default, and under "Program/script," fill in the filepath of the Python file that you want to run (not core.py, your second one). Then press "OK."
  5. Go to the "Settings" tab and uncheck the box that says "Stop the task if it runs longer than 3 days." Then press "OK" and enter your computer's password.

And that's it. You now have everything that you need in order to run custom Python bot scripts on Wikia. If you are running a script that needs to operate continually, such as discussions_delete, it will only run when your PC is on.

Note that the code is optimized for Windows 10. It will not work on Linux or OSX without modifications to core.py; specifically, the way that the script calls on the read/write file for login. On Linux, you should import os instead of pathlib and the first parameter for open should be something more like os.environ['HOME']+'/.discussions-bot/session.p'. If you're on OSX, you're on your own here.

Common errors[]

Error Fix
AttributeError: 'NoneType' object has no attribute 'get' This is caused when the get() function is being used on something that does not exist, i.e. if a session has not been created but is still being called. If you experience this error, there's something wrong with the login function in core.py, probably the try/catch clause it uses to check if it has an existing session. Depending on how it's set up, recent changes to the bot's password or login file may also interact with or cause this error.
Last run result: (0x1)
AttributeError: 'NoneType' object has no attribute 'casefold' The casefold() function is used to decapitalize scanned strings. Check the JSON of your API call and see if you've indexed the specified variable incorrectly. Try printing the index path and see how widespread this is; if it only happens for a single post, it's probably a server-side issue. In that case, as long as the update_interval value is low, the bot will move past it. If it happens for all posts, your indexing is broken.
EOFError: Ran out of input This error may occur if the bot's password has been changed, but not updated in the files, and the operator then updates the login file still without updating the password. It may also occur if the username/password has been updated in the files, but is not correct. If you fix the login credentials, it should work.
socket.gaierror: [Errno 11001] getaddrinfo failed You probably don't have an internet connection.
_pickle.UnpicklingError: invalid load key, 'x'. This error may occur if for some reason you put some text of your own into the login file (after creating a new one, perhaps). Create a new empty login file and you will not get it anymore.
ModuleNotFoundError: No module named 'core' If you're getting this error, you're trying to import a file named core that doesn't exist in the same directory (folder) as the current file. Put both core.py and discussions_delete.py in the same folder and try again.
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape. This means that Python is having trouble reading the filepath you've written in core.py. If you used any backslashes \ in your filepath, replace them with forward slashes /.
The operator or administrator has refused the request (0x800710E0) This sometimes happens when the bot runs for a while and eventually fails. Notice that the task "Status" is probably set to "Running" all the time, when it should really be set to "Ready." You need to manually end and restart the task. Ideally, you should also change your settings to have it automatically end and restart after a certain amount of time.
KeyError: 'query' This means that the API data you're trying to access does not exist. Your query, probably related to login information, is not specified correctly. Review the wiki's MediaWiki API documentation or the JSON of the Discussions API to figure out what's wrong with the path you've provided.
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 74-74: Non-BMP character not supported in Tk This error occurs when IDLE is unable to read a character that you're trying to print out, such as an emoji in a thread title. It breaks everything in the script after the source of the problem.
<Response [400]> This is an HTTP response status code signifying that the user agent has made a request containing "malformed syntax" of some sort. The request probably has to be changed in order to work properly.
If you print the JSON of the HTTP request and get something like {'error': 'pow', 'error_description': 'Missing pow counter headers'}, the issue is that the request is missing some "proof of work" information necessary for authentication. To get around this, you need to use the X-Wikia-WikiaAppsId header in your login file, core.py. It can contain anything, such as 1234.
<Response [401]> This is an HTTP response status code signifying that the user agent is unauthorized (unauthenticated). Authenticate your bot account properly in login to avoid this.
<Response [403]> This is an HTTP response status code signifying that the user agent is trying to do something forbidden by its permissions. You may receive this error if you fail to log into an account with elevated permissions and then try to use said permissions, or if the account you log into does not have the required permission.
<Response [429]> This is an HTTP response status code signifying "too many requests" to the server. Slow the bot down.

JSON[]

Here is the decoded json() output of a post containing some text and a hypertext link. Notice the API endpoints that have been assigned to variables in the code above (spaced awkwardly, but it's all there). Certain terms in the dictionary are nested and must therefore be accessed through other terms (e.g. post['creationDate']['epochSecond']).

{"_links": {"self": {"href": "/1619358/posts/4400000000007534194"}, "permalink": [{"href": "/1619358/permalinks/posts/4400000000007534194?viewableOnly=true"}], "up": [{"href": "/1619358/threads/4400000000001459693"}]}, "createdBy": {"id": "32045548", "avatarUrl": "https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453", "name": "AkulakhanBot", "badgePermission": "badge:threadmoderator"}, "creationDate": {"nano": 0, "epochSecond": 1601912600}, "creatorId": "32045548", "creatorIp": "", "forumId": "1619358", "forumName": "General", "funnel": "TEXT", "id": "4400000000007534194", "isDeleted": false, "isEditable": true, "isLocked": false, "isReply": false, "isReported": false, "jsonModel": "{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}", "latestRevisionId": "4400000000007993868", "modificationDate": null, "position": 1, "rawContent": "Unlinked text linked text unlinked text", "renderedContent": null, "requesterId": "32045548", "siteId": "1619358", "threadCreatedBy": {"id": "32045548", "avatarUrl": "https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453", "name": "AkulakhanBot", "badgePermission": "badge:threadmoderator"}, "threadId": "4400000000001459693", "title": "Field test", "upvoteCount": 0, "_embedded": {"contentImages": [], "userData": [{"hasReported": false, "hasUpvoted": false, "permissions": ["canUndelete", "canMove", "canModerate", "canUnlock", "canEdit", "canDelete", "canLock"]}], "attachments": [{"atMentions": [], "contentImages": [], "openGraphs": [], "polls": [], "quizzes": []}], "thread": [{"containerId": "1619358", "containerType": "FORUM", "creatorId": "32045548", "firstPost": {"id": "4400000000007534194", "renderedContent": null, "jsonModel": "{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}", "createdBy": {"id": "32045548", "avatarUrl": "https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453", "name": "AkulakhanBot", "badgePermission": "badge:threadmoderator"}, "title": "Field test", "attachments": {"openGraphs": [], "contentImages": [], "polls": [], "quizzes": [], "atMentions": []}, "threadId": "4400000000001459693", "createdByIp": null}, "isEditable": true, "isFollowed": true, "isLocked": false, "isReported": false, "postCount": "0", "tags": [], "title": "Field test"}], "latestRevision": [{"creationDate": {"nano": 0, "epochSecond": 1601912600}, "creatorId": "32045548", "creatorIp": "", "id": "4400000000007993868", "jsonModel": "{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}", "postId": "null", "rawContent": "Unlinked text linked text unlinked text", "renderedContent": null}]}}
Formatted JSON
{
   "_links":{
      "self":{
         "href":"/1619358/posts/4400000000007534194"
      },
      "permalink":[
         {
            "href":"/1619358/permalinks/posts/4400000000007534194?viewableOnly=true"
         }
      ],
      "up":[
         {
            "href":"/1619358/threads/4400000000001459693"
         }
      ]
   },
   "createdBy":{
      "id":"32045548",
      "avatarUrl":"https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453",
      "name":"AkulakhanBot",
      "badgePermission":"badge:threadmoderator"
   },
   "creationDate":{
      "nano":0,
      "epochSecond":1601912600
   },
   "creatorId":"32045548",
   "creatorIp":"",
   "forumId":"1619358",
   "forumName":"General",
   "funnel":"TEXT",
   "id":"4400000000007534194",
   "isDeleted":false,
   "isEditable":true,
   "isLocked":false,
   "isReply":false,
   "isReported":false,
   "jsonModel":"{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}",
   "latestRevisionId":"4400000000007993868",
   "modificationDate":null,
   "position":1,
   "rawContent":"Unlinked text linked text unlinked text",
   "renderedContent":null,
   "requesterId":"32045548",
   "siteId":"1619358",
   "threadCreatedBy":{
      "id":"32045548",
      "avatarUrl":"https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453",
      "name":"AkulakhanBot",
      "badgePermission":"badge:threadmoderator"
   },
   "threadId":"4400000000001459693",
   "title":"Field test",
   "upvoteCount":0,
   "_embedded":{
      "contentImages":[
         
      ],
      "userData":[
         {
            "hasReported":false,
            "hasUpvoted":false,
            "permissions":[
               "canUndelete",
               "canMove",
               "canModerate",
               "canUnlock",
               "canEdit",
               "canDelete",
               "canLock"
            ]
         }
      ],
      "attachments":[
         {
            "atMentions":[
               
            ],
            "contentImages":[
               
            ],
            "openGraphs":[
               
            ],
            "polls":[
               
            ],
            "quizzes":[
               
            ]
         }
      ],
      "thread":[
         {
            "containerId":"1619358",
            "containerType":"FORUM",
            "creatorId":"32045548",
            "firstPost":{
               "id":"4400000000007534194",
               "renderedContent":null,
               "jsonModel":"{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}",
               "createdBy":{
                  "id":"32045548",
                  "avatarUrl":"https://static.wikia.nocookie.net/ba0e6cc9-3027-4893-9d58-a1bb2bf0b453",
                  "name":"AkulakhanBot",
                  "badgePermission":"badge:threadmoderator"
               },
               "title":"Field test",
               "attachments":{
                  "openGraphs":[
                     
                  ],
                  "contentImages":[
                     
                  ],
                  "polls":[
                     
                  ],
                  "quizzes":[
                     
                  ],
                  "atMentions":[
                     
                  ]
               },
               "threadId":"4400000000001459693",
               "createdByIp":null
            },
            "isEditable":true,
            "isFollowed":true,
            "isLocked":false,
            "isReported":false,
            "postCount":"0",
            "tags":[
               
            ],
            "title":"Field test"
         }
      ],
      "latestRevision":[
         {
            "creationDate":{
               "nano":0,
               "epochSecond":1601912600
            },
            "creatorId":"32045548",
            "creatorIp":"",
            "id":"4400000000007993868",
            "jsonModel":"{\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"Unlinked text \"},{\"type\":\"text\",\"marks\":[{\"type\":\"link\",\"attrs\":{\"href\":\"http://www.republiquedesmangues.fr/\",\"title\":null}}],\"text\":\"linked text\"},{\"type\":\"text\",\"text\":\" unlinked text\"}]}]}",
            "postId":"null",
            "rawContent":"Unlinked text linked text unlinked text",
            "renderedContent":null
         }
      ]
   }
}


Advertisement