https://github.com/leuc/acp-127
Allied Communications Publications 127g "TAPE RELAY PROCEDURES" for the Central Foreign Policy Files, 1973-1979, Released Telegrams, Record Group 59: General Records of the Department of State, National Archives
https://github.com/leuc/acp-127
cablegate cfpf data-science diplomatic-cables
Last synced: about 1 month ago
JSON representation
Allied Communications Publications 127g "TAPE RELAY PROCEDURES" for the Central Foreign Policy Files, 1973-1979, Released Telegrams, Record Group 59: General Records of the Department of State, National Archives
- Host: GitHub
- URL: https://github.com/leuc/acp-127
- Owner: leuc
- License: gpl-3.0
- Created: 2026-05-30T10:43:43.000Z (about 2 months ago)
- Default Branch: main
- Last Pushed: 2026-05-30T13:59:54.000Z (about 2 months ago)
- Last Synced: 2026-05-30T15:20:22.782Z (about 2 months ago)
- Topics: cablegate, cfpf, data-science, diplomatic-cables
- Language: Python
- Homepage:
- Size: 140 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
- Agents: AGENTS.md
Awesome Lists containing this project
README
A Python rebulk parser for Allied Communications Publications (ACPs) 127 (G),
"TAPE RELAY PROCEDURES" Telegrams from the Central Foreign Policy Files,
1973-1779, Released Telegrams.
Record Group 59: General Records of the Department of State, National Archives.
# Data Source:
```
https://archive.org/download/U.s.DiplomaticCablesYear1973/us-diplomatic-cables-txt-1973.7z
https://archive.org/download/U.s.DiplomaticCablesYear1974/us-diplomatic-cables-txt-1974.7z
https://archive.org/download/U.s.DiplomaticCablesYear1975/us-diplomatic-cables-txt-1975.7z
https://archive.org/download/U.s.DiplomaticCablesYear1976/us-diplomatic-cables-txt-1976.7z
https://archive.org/download/U.s.DiplomaticCablesYear1977/us-diplomatic-cables-txt-1977.7z
https://archive.org/download/U.s.DiplomaticCablesYear1978/us-diplomatic-cables-txt-1978.7z
https://archive.org/download/U.s.DiplomaticCablesYear1979/us-diplomatic-cables-txt-1979.7z
```
# Installation
## Ubuntu Dependencies
```bash
sudo apt-get update
sudo apt-get install python3-orjson python3-rebulk
```
## Using pip (alternative or supplement)
```bash
pip install -r requirements.txt
```
# Downloading and Extracting Data
```bash
# Create a directory for the data
mkdir -p cables
cd cables
# Download and extract each year (1973-1979)
for year in {1973..1979}; do
echo "Downloading $year..."
curl -LO "https://archive.org/download/U.s.DiplomaticCablesYear$year/us-diplomatic-cables-txt-$year.7z"
echo "Extracting $year..."
7z x us-diplomatic-cables-txt-$year.7z
# Optional: remove the archive to save space
rm us-diplomatic-cables-txt-$year.7z
done
```
# Usage
## Basic Usage
Process all .txt and .tel files in a directory (recursively):
```bash
python3 -m src.extractor cables/
```
## With Limits
Limit the number of files processed (useful for testing):
```bash
python3 -m src.extractor cables/ --limit 100
```
## Random Sampling
Process a random sample of files (overrides --limit if both are given):
```bash
python3 -m src.extractor cables/ --sample 1000
```
## Output Format
- **STDOUT**: Each processed file's extraction result is output as a separate JSON line (streaming)
- **STDERR**: Progress messages and final coverage statistics (as a JSON object)
Example of processing a single file and viewing the output:
```bash
# Process one file, capture stdout and stderr separately
python3 -m src.extractor cables/1973/01/1973LIMA00001.txt 2>coverage.json >result.json
# View the extraction result
cat result.json | jq .
# View the coverage statistics
cat coverage.json | jq .
```
## Batch Processing (Multiple Years)
The script uses multiprocessing internally (`ProcessPoolExecutor`), so a simple sequential loop is all that's needed:
```bash
mkdir -p results
for year in {1973..1979}; do
echo "Processing $year ..."
python3 -m src.extractor cables/$year/ 2>results/$year-coverage.json >results/$year-results.ndjson
count=$(wc -l < "results/$year-results.ndjson")
echo "Finished $year: $count records"
done
```
# Reference Normalization Pipeline
Extract structured reference data from ACP-127 messages and build a directed reference graph for network analysis.
## Pipeline
```bash
# 1. Extract structured JSON from ACP-127 messages
python3 -m src.extractor
# 2. Flatten to reftel NDJSON (extract relevant fields for faster loading)
for year in {1973..1979}; do
jq -Mc '{"references": ._reference, "attr_reference": ."Message Attributes"."Reference",
"document_number": ."Message Attributes"."Document Number",
"date": ."Message Attributes"."Draft Date" // ."Message Attributes"."Sent Date",
"message_preview": (._message_content | if . then split("\n")[:100] | join("\n") else null end)}' \
${year}.ndjson > ${year}.reftel.ndjson
done
# 3. Normalize references to canonical MRN format
python3 -m src.reftel_normalize *.reftel.ndjson > all-mrns.ndjson
# 4. Build reference graph (GraphML)
python3 src/reftel2graph.py all-mrns.ndjson reference-graph.graphml
# 5. Analyze graph connectivity
python3 src/analyze_graph.py reference-graph.graphml
```
## Key Files
| File | Description |
|---|---|
| `src/reftel_normalize.py` | Reference normalizer: rebulk functional pattern, O(1) station dict |
| `src/reftel2graph.py` | GraphML builder from normalized NDJSON |
| `src/analyze_graph.py` | Graph analysis: WCC, k-core, PageRank, communities |
| `src/station_data.py` | 558 canonical stations + 686 variant mappings |
See `AGENTS.md` and `docs/REFTEL.md` for detailed architecture and failure analysis.
# LICENSE
[](https://www.gnu.org/licenses/gpl-3.0)