Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/xdevplatform/compliant-client

A set of Python scripts for the Tweet and User batch compliance endpoints. Includes an app that manages it all in one go.
https://github.com/xdevplatform/compliant-client

python twitter twitter-api twitter-api-v2 twitter-compliance

Last synced: about 2 months ago
JSON representation

A set of Python scripts for the Tweet and User batch compliance endpoints. Includes an app that manages it all in one go.

Host: GitHub
URL: https://github.com/xdevplatform/compliant-client
Owner: xdevplatform
License: apache-2.0
Created: 2020-12-04T06:10:05.000Z (about 4 years ago)
Default Branch: main
Last Pushed: 2021-09-13T15:42:00.000Z (over 3 years ago)
Last Synced: 2024-08-18T16:16:08.532Z (4 months ago)
Topics: python, twitter, twitter-api, twitter-api-v2, twitter-compliance
Language: Python
Homepage: https://developer.twitter.com/en/docs/twitter-api/compliance/batch-compliance/introduction
Size: 142 KB
Stars: 14
Watchers: 13
Forks: 4
Open Issues: 1
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

# The Compliant (Python) client

**tl;dr** This repository provides both simple scripts and a more elaborate *client* for working with the "batch
Compliance" endpoints. There is a simple script for each of the 5 stages of managing a Job life-cycle, while the
client automates a Compliance Job's life-cycle, from creating a Job to downloading the results.

+ [Introduction](#intro)
+ [Simple scripts](#simple)
+ [Example client](#client)
+ [Getting Started](#starting)
+ [Python packages](#packages)
+ [Setting up authentication](#auth)
+ [Compliance Job attributes and lifecycles](#lifecycles)
+ [Example ID file](#id-file)
+ [Simple scripts](#simple-details)
+ [Example client](#client-details)
+ [Example calls](#example-calls)
+ [Compliance class](#compliance-class)
+ [Core objects](#core-objects)

## Introduction

This repository contains Python code for working with the Twitter Batch Compliance endpoints. To learn more about these
endpoints, check out the [Batch compliance documentation](https://developer.twitter.com/en/docs/twitter-api/compliance/batch-compliance/introduction).

### Simple scripts
This repository hosts a set of simple Python scripts (in the ./scripts folder) for the Compliance endpoints. These
scripts support a simple command line interface.

These five scripts map to the fundamental methods that are called on the new Compliance endpoint:
1) Create a Compliance Job: ```create_job.py```
2) Once Job is created, upload a set of Twitter Tweet IDs or User IDs to check for Compliance events
related to them: ```upload_ids.py```
3) Request the status of a Job. Determine if a Job was created, whether it is in progress or completed: ```list_job.py```
4) Request the status of all "active" Jobs: ```list_jobs.py```
5) Once a Job has the status of 'completed', download the results that indicate which Tweets have been deleted, which
User accounts have updates, or some other Compliance event such as geo-scrubbing: ```download_results.py```

**Some important notes!**
* Only one Job at a time can be running, and uploading an ID file triggers the start of procoessing.
* The provided upload URL expires in 15 minutes. If the upload URL expires before you get the IDs uploaded, you will
need to create a new Job.
* The download URL expires one week after the Job is created. Be sure to download all of your results within 7 days
of creating the Job.

### Example client

There is also a more elaborate example 'compliant-client' script that helps manage Compliance Jobs and their lifecycles.
For example, the 'apps/compliant-client.py' script enables you to manage Jobs by name and not just ID. More interestingly,
this client code lets you provide all that is needed in one "all" command. The script creates a new Job, uploads
a specified User ID or Tweet ID file, checks on the Job status every 30 seconds, then downloads the results when the Job completes.

When making an 'all' command, you need to specify the Job type, a Job name, the path to the IDs file for uploading, and the path to
file where you want the results written. The Batch Compliance endpoints support two types of Compliance requests, one for Tweet Compliance
events, and one for User Compliance events.

Here is an example command-line for creating a Job, uploading a Tweet ID file, monitoring a Job's progress, and writing the downloaded results to a local file:

```bash
$python apps/compliant-client.py --all --type tweets --name "Checking Tweets" --ids-file "../inbox/tweet_ids.txt" --results-file "../outbox/tweet_results.json"
```
Here's an example client command-line for working with a User ID file:

```bash
$python apps/compliant-client.py --all --type users --name "Checking Users" --ids-file "../inbox/user_ids.txt" --results-file "../outbox/user_results.json"
```

## Getting started

The first step when starting with the batch Compliance endpoint is establishing access and generating the authentication
token needed to make requests. You will need to have an approved developer account and have access to the Twitter
Developer Portal. You will need to have a Twitter App, and have its Bearer Token on-hand to start making requests. Bearer
Tokens can be generated in the Developer Portal (yay!). Authentication tokens are only displayed on the Developer Portal
once (when generated), so be ready to save those tokens in a secure place.

The simple scripts and the example client share common code that imports your Bearer Token from the local os
environment/session (this client was developed on the MacOS. If you are on Windows you may want to update how Tokens are
managed). See the "Setting up authentication" section below for more details.

So far, a package has not been generated for this project. To start working with the client code, the repository can be
cloned in your environment. Currently, there is not any configuration file needed, and the only 'configuration' needed is
setting up of a "BEARER_TOKEN" environment variable.

### Python packages
The scripts and example app import the following Python Packages:
* requests
* json
* os
* docopt (A package that converts simple header documentation into a set of command-line parameters, options, and switches.)

... and that's it!

Other steps for preparing to use the batch Compliance endpoint include:

+ Preparing Tweet and User ID files. IDs should be in simple text with one ID per line. When uploading
the files, a 'Content-Type' header should be set to 'text/plain'. See below for an example.
+ Build methods to import and update Tweet archives. Once the Compliance results are downloaded, code is needed to update
your archives accordingly. This may take the form of making database deletes, deleting stored JSON files, or
writing new file data files.

### Setting up authentication

To set your enviornment variables in your terminal run the following lines to load your tokens into these environmental
variables. Note that there variables are session-specific, so if you stop and restart a terminal these will need to be
reloaded unless you auto-load them or persist them somehow.

```bash
export 'BEARER_TOKEN'=''
```

A quick way to test your authentication is to make a request for the current "list Jobs."

```bash
$python ./scripts/list_jobs.py --type tweets
```

```bash
$python ./apps/compliant-client.py --list --type tweets
```

The example scripts and example client code includes this common code that loads these tokens in from the local environment:

```python
import requests
import os

def bearer_oauth(self, r):
# To set your environment variables in your terminal run the following line:
# export 'BEARER_TOKEN'=''
bearer_token = os.environ.get("BEARER_TOKEN")
r.headers['Authorization'] = "Bearer {}".format(bearer_token)

return r

URL = 'https://api.twitter.com/2/compliance/jobs'
headers = {}
response = requests.get(f"{URL}/{id}", auth=bearer_oauth, headers=headers)

```

## Job attributes and lifecycles

Compliance Jobs have a lifecycle, from being created, having Twitter IDs assigned, a period of time while the results
are generated, to having a file to download that contains JSON that describes Compliance events associated with submitted IDs.

When a Job is created, URLs for uploading IDs and downloading results from are provided. When Twitter IDs are uploaded,
the process of checking each ID for Compliance events starts. This process will take many minutes (depending on the
number of IDs submitted). After the job completes, the results can be downloaded for post-processing. The results have the
form of JSON that describes the type of Compliance event that took place.

Let's dig into some of the details...

* Job are created with an optional (and recommended) name.

The 'create' process creates a new job with the following
attributes:

```json
{
"created_at": "2021-08-25T00:21:54.000Z",
"download_expires_at": "2021-09-01T00:21:54.000Z",
"download_url": "https://storage.googleapis.com/twttr-tweet-compliance/1430324519596564483/delivery/992443111073660928_1430324519596564483?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=complianceapi-public-svc-acct%40twttr-compliance-public-prod.iam.gserviceaccount.com%2F20210825%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210825T002154Z&X-Goog-Expires=604800&X-Goog-SignedHeaders=host&X-Goog-Signature=4066eea9ff73d297d5f51ac6f1260540bd8656b473f33e2cc0925afe9c58c8bc66c6c8a501a059ee90bc3c78144c1b32ea774660a3672f33e44c0083f2c9da3ff7fcf6efd8499e158733b81fb9ad9e3f91144d988d903404f5e3718f37719627014fe34b1b4fd7af41ca76393e0a6564cdd03b1cb1aa3d385a639940f9e52f6b5f3d9e5d023a55811bbbed6e5b54e077a2d413a392afd5088b3906ce3d40162f4ca4f62b26e363f70a57ea252594b091e6a50480753f9a528f6bd1c7890883b66b55750220e45fff8aa64067cf08c390517b597d4d80d759c124be5352446e57f9b57dcf3c070bee913cd6140897485a6eabd92157faee46f857f1939931c0f9",
"id": "1430324519596564483",
"name": "Checking my stored Tweets",
"resumable": false,
"status": "created",
"type": "tweets",
"upload_expires_at": "2021-08-25T00:36:54.000Z",
"upload_url": "https://storage.googleapis.com/twttr-tweet-compliance/1430324519596564483/submission/992443111073660928_1430324519596564483?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=complianceapi-public-svc-acct%40twttr-compliance-public-prod.iam.gserviceaccount.com%2F20210825%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210825T002154Z&X-Goog-Expires=900&X-Goog-SignedHeaders=content-type%3Bhost&X-Goog-Signature=c20ecee2fea6e4ada2cb04f08b3350b039d9a608dd343b9c1ac2a003d2c26e6abac7546a5589bc3146d87b3e9e6023309c96bb9cddf96f8281e2a17fa63fdd8f711a997317c4be6dfead08d728b48381f4d95e515d01b91066aa4eb0c876bcd12195d2521ec2720c8e92d2830ae4ef17340c4fb92c58203f05bef9474d8d91404f48eff41b3b04adb662668eecd14b4ccc848d4d0e7df690a48e5befca081e0ce429a4387ba763a47bec7e88a3a57fdb2b97f7a2a04294aaafe102d9704dce62967e2a85c6c3170c689a8ba8f079afaf63ba850846aadd4ef7a99a7f67e0c05bf65a628809af79251b31051e8a785e262c28b81f2b3aa2839adaf1f5d078e9c2"
}
```
These Job details include an URL for uploading your ID. There is also an expiration time for that URL. Once a Job is
created the upload link is available for 15 minutes. If the link expires before you upload the IDs, a new Job will need
to be created.

* Tweet and User IDs are uploaded to a cloud file system. The IDs are written to a simple text file with one ID per line.
Below is an example for a set of Tweet IDs, and the format is identical for User Ids.

#### Tweet ID file

```
900145569397649408
900145765250732032
900148142313754624
900148877004861440
900149201455263746
900149551985819648
900149689252802560
900149981465673732
900150802534342657
900152816081219584
```

* Uploaded IDs are checked for Compliance, a process that can take many minutes (and hours?) to complete.

```json
{
"status": "in_progress"
}

```

The Job will remain in the "in_progress" status until it finishes and enters a final "completed" status

```json
{
"created_at": "2021-08-25T00:21:54.000Z",
"download_expires_at": "2021-09-01T00:21:54.000Z",
"download_url": "https://storage.googleapis.com/twttr-tweet-compliance/1430324519596564483/delivery/992443111073660928_1430324519596564483?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=complianceapi-public-svc-acct%40twttr-compliance-public-prod.iam.gserviceaccount.com%2F20210825%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210825T002154Z&X-Goog-Expires=604800&X-Goog-SignedHeaders=host&X-Goog-Signature=4066eea9ff73d297d5f51ac6f1260540bd8656b473f33e2cc0925afe9c58c8bc66c6c8a501a059ee90bc3c78144c1b32ea774660a3672f33e44c0083f2c9da3ff7fcf6efd8499e158733b81fb9ad9e3f91144d988d903404f5e3718f37719627014fe34b1b4fd7af41ca76393e0a6564cdd03b1cb1aa3d385a639940f9e52f6b5f3d9e5d023a55811bbbed6e5b54e077a2d413a392afd5088b3906ce3d40162f4ca4f62b26e363f70a57ea252594b091e6a50480753f9a528f6bd1c7890883b66b55750220e45fff8aa64067cf08c390517b597d4d80d759c124be5352446e57f9b57dcf3c070bee913cd6140897485a6eabd92157faee46f857f1939931c0f9",
"id": "1430324519596564483",
"name": "Checking my stored Tweets",
"resumable": false,
"status": "complete",
"type": "tweets",
"upload_expires_at": "2021-08-25T00:36:54.000Z",
"upload_url": "https://storage.googleapis.com/twttr-tweet-compliance/1430324519596564483/submission/992443111073660928_1430324519596564483?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=complianceapi-public-svc-acct%40twttr-compliance-public-prod.iam.gserviceaccount.com%2F20210825%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20210825T002154Z&X-Goog-Expires=900&X-Goog-SignedHeaders=content-type%3Bhost&X-Goog-Signature=c20ecee2fea6e4ada2cb04f08b3350b039d9a608dd343b9c1ac2a003d2c26e6abac7546a5589bc3146d87b3e9e6023309c96bb9cddf96f8281e2a17fa63fdd8f711a997317c4be6dfead08d728b48381f4d95e515d01b91066aa4eb0c876bcd12195d2521ec2720c8e92d2830ae4ef17340c4fb92c58203f05bef9474d8d91404f48eff41b3b04adb662668eecd14b4ccc848d4d0e7df690a48e5befca081e0ce429a4387ba763a47bec7e88a3a57fdb2b97f7a2a04294aaafe102d9704dce62967e2a85c6c3170c689a8ba8f079afaf63ba850846aadd4ef7a99a7f67e0c05bf65a628809af79251b31051e8a785e262c28b81f2b3aa2839adaf1f5d078e9c2"
}
```

* After the job is 'completed', the results can be downloaded from a cloud file service.

```json
{
"download_expires_at": "YYYY-MM-DDTHH:mm:ss.000Z",
"download_url": "https://storage.googleapis.com/twttr-tweet-compliance/12345678888888/delivery/..."
}
```

## Compliance results objects

Currently, the batch compliance endpoints support the following compliance events:
+ Tweet and User Deletes (a Twitter user deletes a Tweet or their account)
+ Tweet Scrub Geos (a Twitter user removes geo data from their Tweets)
+ Tweet and User Protects (a Twitter user protects their Tweets, ie “goes private”)
+ Tweet and User Suspensions (a Twitter user is suspended, and their Tweets and account are no longer visible on the platform)

Here is an example for a Tweet Compliance event:

```json
{
"id": "906972198136631298",
"action": "delete",
"created_at": "2017-09-10T20:06:37.421Z",
"redacted_at": "2020-07-21T23:37:55.607Z",
"reason": "deleted"
}
```

Here is an example for a User Compliance event:

```json
{
"id": "#######",
"action": "delete",
"created_at": "2013-03-26T02:46:15+00:00",
"reason": "protected"
}
```

Other reasons include: "deleted" and "suspended"

## Simple scripts
##### ./scripts

The /scripts folder contains a set of Python scripts for the compliance endpoints. These scripts can be used to create
and manage Compliance Jobs and their lifecycles:

These scripts all support command-line arguments for passing in settings when creating a Job, or when listing Jobs. Here
is an example for the list_job.py script.

```python
if __name__ == "__main__":

arguments = docopt(__doc__, version='v1.0')

job_details = list_job(arguments['--id'])

if len(job_details) == 0:
print(f"Compliance Job not found or error occurred.")
else:
print(f"Job details for ID {job_details['id']}:")
print(json.dumps(job_details, indent=4, sort_keys=True))
```
There are five scripts:
1) **create_job.py** Creating a Job: ```python create_job.py --type tweets --name "My new Compliance Job"```
2) **list_jobs.py** Requesting a list of all existing Jobs: ```python list_jobs.py --type tweets```
3) **list_job.py** Checking on a specific Job status and retrieving its details: ```python list_jobs.py --id ```
4) **upload_ids.py**Uploading a text file with Tweet IDs (one per line): ```python upload_ids.py --ids-file --url ```
5) **download_results.py** Downloading results which consist of one JSON object for each Tweet that has had Compliance event (e.g. has been
deleted): ```python download_results.py --id --url ```

## Example client
##### ./apps/compliant-client.py
+ Command-line app for working with the Twitter API v2 compliance endpoint.

The commannd-line supports the following commands (these are displayed with the -h or --help command):

```
compliant-client.py

Usage:
compliant-client --all --type --name --ids-file --results-file
compliant-client --create --type --name
#Client will return both Tweet and User jobs if type is not specified.
compliant-client --list [--type | --name | --id | --status ]
compliant-client --upload (--name | --id ) --ids-file
compliant-client --download (--name | --id ) --results-file
compliant-client --help
compliant-client --version

Options:
-a --all
-c --create
-l --list
-s --status STATUS
-u --upload
-d --download
-t --type TYPE
-n --name NAME
-i --id ID
-f --ids-file IDSFILE
-r --results-file RESULTSFILE
-h --help
-v --version
```

### Example calls

Here are some example commands, and illustrate how to manage a Job from creation to downloading results.

This will create a new Tweets compliance Job, and apply the name "MyTweetsJob." Assigning a name will enable you to
manage Jobs by that name instead of its ID. That can be a nice convenience.
```bash
$python compliant-client.py --create --type tweets --name "Checking my stored Tweets"
```

This will list the status of all Tweets Jobs.
```bash
$python compliant-client.py --list --type tweets
```

You can also look up a Job status by name.
```bash
$python compliant-client.py --list --name "Checking my stored Tweets"
```

This command will upload the Tweet IDs for the Job named "MyTweetsJob".
```bash
$python compliant-client.py --upload --type tweets --name "Checking my stored Tweets" --ids-file "./inbox/tweet_ids.txt"
```

When the Job is finished, this will download the results for the Job named "MyTweetsJob" to the file specified.
```bash
$python compliant-client.py --download --type tweets --name "Checking my stored Tweets" --results-file "./oubox/results.txt"
```

**If you want to make a single 'all' call**, the script will manage the Job from creation to downloading the results:
```bash
$python compliant-client.py --all --name "Checking my stored Tweets" --ids-file "../inbox/tweet_ids.txt" --results-file "../outbox/results.json"
```

### Compliance-client class

#### compliant-client/compliance/compliance.py

The *compliance.py* class lives in a *../compliance folder*. The *../compliance/compliance.py* file defines a
**compliance_client** class.

```python
import compliance.compliance

compliance_client = compliance.compliance.compliance_client() # Geez, that some odd looking syntax.

job_details = compliance_client.create_compliance_job(settings['name'])
results = compliance_client.upload_ids(settings['ids-file'], job_details['upload_url'])
results = compliance_client.download_results(job_details['download_url'], settings['results-file'])
current_jobs = compliance_client.list_jobs()
job_details = compliance_client.list_job(settings['id'])

```

### Core objects

#### Job details

This code works with a **job_details** object. When you call the 'create job' method of the Compliance class, it will return
a ```job_details``` object with all the attributes you need to manage the Job's lifecycle.

```python
job_details = compliance_client.create_compliance_job(type, name)
```
The Batch compliance endpoint returns a JSON object with several attributes:

```json
{
"download_expires_at": "2020-12-30T00:05:30.000Z",
"download_url": "https://storage.googleapis.com/twttr-tweet-compliance/1341535366134689792/delivery/...",
"id": "1341535366134689792",
"name": "80M Tweet test",
"status": "complete",
"type": "tweets",
"upload_expires_at": "2020-12-23T00:20:30.000Z",
"upload_url": "https://storage.googleapis.com/twttr-tweet-compliance/1341535366134689792/submission/..."
}
```

The Python dictionary that encapsulates Job objects looks like:

```python

job_details = {}
job_details['name']
job_details['type']
job_details['id']
job_details['upload_url']
job_details['download_url']
job_details['status']
job_details['upload_expires_at']
job_details['download_expires_at']

```