{"id":44512032,"url":"https://github.com/rootsdev/nama","last_synced_at":"2026-02-13T15:21:36.820Z","repository":{"id":45952241,"uuid":"417523893","full_name":"rootsdev/nama","owner":"rootsdev","description":"use deep learning to find similar personal names","archived":false,"fork":false,"pushed_at":"2024-01-25T18:01:24.000Z","size":12549,"stargazers_count":1,"open_issues_count":7,"forks_count":0,"subscribers_count":7,"default_branch":"master","last_synced_at":"2024-03-26T07:23:48.511Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rootsdev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-10-15T14:12:20.000Z","updated_at":"2023-02-27T20:32:42.000Z","dependencies_parsed_at":"2023-01-29T03:31:08.153Z","dependency_job_id":"54ec942f-ef8e-4697-a198-23cbd92744e4","html_url":"https://github.com/rootsdev/nama","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/rootsdev/nama","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsdev%2Fnama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsdev%2Fnama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsdev%2Fnama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsdev%2Fnama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rootsdev","download_url":"https://codeload.github.com/rootsdev/nama/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rootsdev%2Fnama/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29411143,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-13T15:21:33.512Z","updated_at":"2026-02-13T15:21:36.712Z","avatar_url":"https://github.com/rootsdev.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# nama\n\nUsing deep learning to find similar personal names\n\n[Presentation Slides](https://docs.google.com/presentation/d/1NFvCRk0fymeCPJqbvHv2S77V_qTTLVZm4bKyZpz5k80/edit#slide=id.g21141600e86_0_6)\n\nstart with old slides,\nfigure out what we want to say and add it here,\nthen create new slides\n\n## Initial Setup\n\n    make install\n\n### If you want to develop, also do the following\n\n    poetry shell\n    nbstripout --install   # automatically strip notebook output before commit\n    pytest                 # run tests\n\n### Notes\n\nYou are now ready to start development on your project!\nThe CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.\n\nFor activating the automatic documentation with MkDocs, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/mkdocs/#enabling-the-documentation-on-github).\nTo enable the code coverage reports, see [here](https://fpgmaas.github.io/cookiecutter-poetry/features/codecov/).\n\n---\n\nRepository initiated with [fpgmaas/cookiecutter-poetry](https://github.com/fpgmaas/cookiecutter-poetry).\n\n## Using nama\n\n### Remote Development\n\n- ssh to the remote server and install python 3.12 and poetry (one time only)\n\n```\nsudo yum update -y\nsudo yum groupinstall -y \"Development Tools\"\nsudo yum install gcc bzip2-devel libffi-devel make perl-core pcre-devel zlib-devel openssl openssl-devel sqlite-devel -y\nwget https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tgz\ntar xzf Python-3.12.2.tgz\ncd Python-3.12.2/\nsudo ./configure --enable-optimizations --enable-loadable-sqlite-extensions\nsudo make altinstall\nsudo ln -s /usr/local/bin/python3.12 /usr/local/bin/python\nsudo ln -s /usr/local/bin/pip3.12 /usr/local/bin/pip\ncurl -sSL https://install.python-poetry.org | python -\n```\n\n- copy your aws config and credentials to the remote server in ~/.aws\n\n- from your local machine, copy the code up to the remote machine\n\n```\nrsync -i nama1.1.pem -av --exclude='\\.*' . ec2-user@[IP]:/home/ec2-user/nama\n```\n\n- ssh to the remote server again and run poetry install and jupyter\n\n```\nssh -i nama1.1.pem -L 8888:localhost:8888 ec2-user@[IP]\ncd nama\npoetry install\npoetry shell\njupyter notebook --no-browser --port=8888\n```\n\n### Notebooks\n\nRun notebooks in the order listed\n\n- 310_clean - clean the raw name pairs from FamilySearch (pairs of tree \u003c-\u003e record name) and separate into given and surnames (2 hours)\n  - input: tree-hr-raw\n  - output: tree-hr-names\n- 311_clean_preferred - generate given and surname preferred names from FamilySearch (30 minutes)\n  - input: pref-names-raw\n  - output: pref-names-interim\n- 320_generate_pairs - generate pairs from best-matching name pieces (15 minutes)\n  - input: tree-hr-names\n  - output: tree-hr-pairs\n- 330_aggregate - aggregate pairs of matching tree \u003c-\u003e record name pieces and compute counts, probabilities, and similarities (15 minutes)\n  - input: tree-hr-pairs\n  - output: frequencies\n- 331_aggregate_preferred - aggregate preferred names (10 minutes)\n  - input: pref-names-interim\n  - output: pref-names\n- 340_filter - convert the tree-hr-attach parquet files into similar and dissimilar name pairs (2 minutes)\n  - input: frequencies\n  - output: similar-names, dissimilar-names\n- 345_train_test_split - split similar names into train and test sets, removing bad pairs (3 minutes)\n  - input: similar-names, pref-names, bad-pairs\n  - output: train, test\n- 360_augment_train_for_swivel - Augment the training dataset with other matching pairs based upon names having the same code or levenshtein similarity (many hours) (DEPRECATED)\n  - input: train\n  - output: train-augments, train-augmented\n- 361_tune_swivel - Run hyperparameter tuning on swivel model (optional - as long as you want to spend) (DEPRECATED)\n  - input: train-augmented\n- 362_train_swivel - Train a swivel model (takes 80 hours for given names, probably 240 hours for surnames) (DEPRECATED)\n  - input: train-augmented (to train), train (to evaluate)\n  - output: swivel-vocab, swivel-model\n- 363_analyze_swivel - Analyze swivel scores and frequencies; determine min_frequency cutoff for generating triplets (takes up to 4 hours) (DEPRECATED)\n  - input: std-buckets, frequencies, swivel-vocab, swivel-model\n- 364_generate_triplets_from_swivel - generate triplets by running swivel over high-frequency names (5 hours) (DEPRECATED)\n  - input: frequencies, swivel-vocab, swivel-model\n  - output: swivel-triplets\n- 370_generate_triplets_for_cross_encoder - generate triplets from training data for the cross encoder (1 hour)\n  - input: train\n  - output: triplets\n- 371_generate_common_non_negatives - generate pairs of names that are not negative examples (\u003c1 hour)\n  - input: std-buckets, pref-names, triplets, given-nicknames\n  - output: common-non-negatives\n- 372_generate_subword_tokenizer - create a subword tokenizer (1 hour)\n  - input: frequencies\n  - output: tokenizer\n- 373_augment_triplets_for_cross_encoder - augment triplets with additional triplets (1 hour)\n  - input: triplets, pref-names, common-non-negatives, tokenizer\n  - output: triplets-augmented\n- 375_train_language_model - train a roberta masked language model in preparation for training the name-pair cross-encoder (2 hours)\n  - input: frequencies\n  - output: roberta\n- 377_train_cross_encoder - train a cross-encoder model (based on sentence bert) that takes a pair of names and outputs a similarity score (32 hours)\n  - input: roberta, triplets-augmented\n  - output: cross-encoder\n- 378_generate_triplets_from_cross_encoder - generate various datasets of triplets for training the bi-encoder from the cross-encoder, cause the bi-encoder needs a lot of data (43 hours)\n  - input: pref-names, train, common-non-negatives, std-buckets, cross-encoder\n  - output: cross-encoder-triplets-train, cross-encoder-triplets-common\n- 380_train_bi_encoder - train a bi-encoder model (1 hour per epoch, so 8 hours for 8 epochs)\n  - input: cross-encoder-triplets-train, cross-encoder-triplets-common, triplets-augmented, tokenizer\n  - output: bi-encoder\n- 381_eval_bi_encoder - evaluate a bi-encoder model (1.5 hours)\n  - input: std-buckets, frequencies, tokenizer, bi-encoder\n- 390_create_clusters_from_buckets - split buckets into clusters using the cross encoder; clusters in the same bucket form a super-cluster (3 hours)\n  - input: std-buckets, tokenizer, cross-encoder, bi-encoder, pref-names\n  - output: clusters, super-clusters\n- 391_augment_clusters - augment clusters with additional names that were not in any cluster (9 hours)\n  - input: basenames, clusters, tokenizer, pref-names, cross-encoder, bi-encoder\n  - output: clusters-augmented\n- 393_compress_clusters - compress cluster and super-cluster files so we can check them into git (1 hour)\n  - input: clusters-augmented, clusters-super\n  - output: clusters-augmented and clusters-super compressed (.gz)\n- 394_eval_coder - compare the precision and recall of nama to familysearch and other coders (3 hours for tiny)\n  - input: clusters-augmented, clusters-super, tokenizer, bi-encoder, train, test, query-names, pref-names, given-nicknames\n- 395_create_phonebook - create the phonebook for surnames (~1 hour)\n  - input: clusters-augmented, clusters-super, pref-names\n  - output: phonebook\n- 396_save_bi_encoder_weights - save the bi-encoder weights so we can use them in fs-nama (java) (1 hour)\n  - input: tokenizer, bi-encoder\n  - output: bi-encoder-weights\n\n### Files\n\n- bad-pairs - pairs of names that are not similar (Clorinda reviewed)\n  - f\"s3://fs-nama-data/2023/familysearch-names/interim/{given_surname}\\_variants_clorinda_reviewed.tsv\"\n- basenames - surnames with prefixes identified\n  - \"../references/basenames-20100616.txt\"\n- bi-encoder - bi-encoder model\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/bi_encoder-ce-{given_surname}-{num_epochs}-{embedding_dim}-{num_epochs}-{bi_encoder_vocab_size}-{learning_rate}.pth\"\n- bi-encoder-weights - bi-encoder weights for fs-nama java code\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/bi_encoder-{given_surname}-{num_epochs}-{embedding_dim}-{num_epochs}-{bi_encoder_vocab_size}-{learning_rate}-weights.json\"\n- clusters - buckets divided into clusters based upon cross-encoder name similarity\n  - f\"s3://fs-nama-data/2024/nama-data/data/processed/clusters\\_{given_surname}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}.json\"\n- clusters-augmented - clusters augmented with additional names using the cross-encoder\n  - f\"s3://fs-nama-data/2024/nama-data/data/processed/clusters\\_{given_surname}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}-augmented.json\"\n- clusters-super - clusters in the same bucket form a super-cluster\n  - f\"s3://fs-nama-data/2024/nama-data/data/processed/super*clusters*{given_surname}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}.json\"\n- common-non-negatives - pairs of common names that may be similar (are not negative)\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/common\\_{given_surname}\\_non_negatives.csv\"\n- cross-encoder - cross-encoder (directory containing multiple files)\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/cross-encoder-{given_surname}-{cross_encoder_vocab_size}/\"\n- cross-encoder-triplets-common - triplets generated from cross-encoder, focusing on negative examples involving common names\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/cross-encoder-triplets-{given_surname}-common.csv\"\n- cross-encoder-triplets-train - triplets generated from cross-encoder\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/cross-encoder-triplets-{given_surname}-train.csv\"\n- dissimilar-names - pairs of names from tree-record attachments that are probably not similar\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-dissimilar.csv.gz\"\n- frequencies - name frequences in hr\n  - f\"s3://fs-nama-data/2024/familysearch-names/interim/tree-hr-{given_surname}-aggr-v2.parquet\"\n- given-nicknames - hand-crafted list of nicknames\n  - \"../references/givenname_nicknames.csv\"\n- phonebook - phonebook for fs-nama\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/{phonebook_type}-phonebook.json\"\n- pref-names - preferred tree names\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- pref-names-interim - preferred tree names before splitting and aggregation\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- pref-names-raw - preferred tree names before separating into given and surname (directory)\n  - f\"s3://fs-nama-data/2024/familysearch-names/raw/tree-preferred/\"\n- query-names - sample of queried names from 2023\n  - f\"s3://fs-nama-data/2023/familysearch-names/processed/query-names-{given_surname}-v2.csv.gz\"\n- roberta - roberta model trained on names (directory containing multiple files)\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/roberta-{given_surname}-{cross_encoder_vocab_size}/\"\n- similar-names - train+test before bad pairs have been removed\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-similar.csv.gz\"\n- std-buckets - the original Steve Blodgett buckets\n  - f\"../references/std\\_{given_surname}.txt\"\n- swivel-triplets - triplets generated by swivel to train the bi-encoder\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-triplets-{hard_neg_count}-{easy_neg_count}.csv.gz\"\n- swivel-model - model generated by swivel\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n- swivel-vocab - vocabulary used by swivel\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n- test - test hr data\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n- tokenizer - (subword) tokenizer\n  - f\"s3://fs-nama-data/2024/nama-data/data/models/fs-{given_surname}-subword-tokenizer-{bi_encoder_vocab_size}.json\"\n- train - training hr data\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n- train-augments - pairs that were added to the training data\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-train-augments.csv.gz\",\n- train-augmented - training hr data augmented with similar names from coders and levenshtein\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\",\n- tree-hr-names - names from tree-record attachments\n  - f\"s3://fs-nama-data/2024/familysearch-names/interim/tree-hr-{given_surname}/\"\n- tree-hr-pairs - pairs of names from tree-record attachments (directory with lots of files)\n  - f\"s3://fs-nama-data/2024/familysearch-names/interim/tree-hr-{given_surname}-pairs/\"\n- tree-hr-raw - tree-record names before they have been split into give and surname (directory)\n  - f\"s3://fs-nama-data/2024/familysearch-names/raw/tree-hr/\"\n- triplets - triplets used to train cross-encoder\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-triplets-{tree_name_min_freq}.csv.gz\"\n- triplets-augmented - augment triplets used to train cross-encoder with similar names according to coders and levenshtein\n\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-triplets-{tree_name_min_freq}-augmented.csv.gz\"\n\n- all-tree-hr-names-sample - 10m sample of all-tree-hr names\n  - f\"../data/processed/all-tree-hr-{given_surname}-sample-10m.txt\"\n- all-tree-hr-names - all tree preferred names\n  - f\"../data/processed/all-tree-hr-{given_surname}.txt\"\n- all-tree-pref-names - all tree preferred names\n  - f\"../data/processed/all-tree-preferred-{given_surname}.txt\"\n- augmented-clusters - similar names from the same bucket that have been augmented with additional frequent names that were not in any bucket\n  - f\"../data/processed/clusters\\_{given_surname}-{scorer}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}-augmented.json\"\n- bad-pairs - pairs of names that are not similar (Clorinda reviewed)\n  - f\"s3://familysearch-names/interim/{given_surname}\\_variants_clorinda_reviewed.tsv\"\n  - I don't recall exactly how the borderline pairs that went to review were generated, but most likely we simply identified similar-v2 training pairs that had low levenshtein similarity. We don't have a notebook for this.\n- bi-encoder-triplets - triplets to train the bi-encoder\n  - f\"s3://fs-nama-data/2024/familysearch-names/processed/tree-hr-{given_surname}-triplets-{hard_negs}-{easy_negs}.csv.gz\"\n- bi-encoder - model to convert a tokenized name to a vector\n  - f\"../data/models/bi_encoder-{given_surname}-{model_type}.pth\"\n- bi-encoder-weights - json file containing bi-encoder token and position weights\n  - f\"../data/models/bi_encoder-{given_surname}-{model_type}-weights.json\"\n- clusters - similar names from the same bucket\n  - f\"../data/processed/clusters\\_{given_surname}-{scorer}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}.json\"\n- common-non-negatives - pairs of names that may be similar (are not negative)\n  - f\"../data/processed/common\\_{given_surname}\\_non_negatives.csv\"\n- cross-encoder - model to evaluate the similarity of two names\n  - f\"../data/models/cross-encoder-{given_surname}-10m-265-same-all\"\n- cross-encoder-triplets-0 - triplets generated from cross-encoder with num_easy_negs=0\n  - f\"../data/processed/cross-encoder-triplets-{given_surname}-0.csv\"\n- cross-encoder-triplets-common - triplets generated from cross-encoder with num_easy_negs='common'\n  - f\"../data/processed/cross-encoder-triplets-{given_surname}-common.csv\"\n- cross-encoder-triplets-common-0-augmented = cross-encoder-triplets-common + cross-encoder-triplets-0 + triplets-augmented\n  - f\"../data/processed/cross-encoder-triplets-{given_surname}-common-0-augmented.csv\"\n- dissimilar-v2 - pairs of names from tree-record attachments that are probably not similar\n  - f\"s3://familysearch-names/processed/tree-hr-{given_surname}-dissimilar-v2.csv.gz\"\n- given-nicknames - nicknames for given names (hand curated from a variety of sources)\n  - f\"../references/givenname_nicknames.csv\"\n- hr-names - names from historical records - Richard provides this by zcat'ing all files into a single file\n  - f\"../data/processed/hr-{given_surname}-aggr.csv.gz\"\n- nearby-clusters - for each cluster, list the nearby clusters\n  - f\"../data/processed/nearby*clusters*{given_surname}-{scorer}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}.json\"\n- phonebook - map surname clusters to partitions\n  - f\"s3://familysearch-names/processed/phonebook.csv\"\n- pref-names - preferred names from the tree\n  - f\"../data/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- pref-names-interim\n  - f\"s3://familysearch-names/interim/tree-preferred-{given_surname}/\"\n- pref-names-raw\n  - f\"s3://familysearch-names/raw/tree-preferred/\"\n- query-names - sample of queried names to be evaluated\n  - f\"../data/processed/query-names-{given_surname}-v2.csv.gz\"\n- roberta - roberta-based language model for names\n  - f\"../data/models/roberta-{given_surname}-10m-{vocab_size}\"\n- similar-v2 - train-v2 + test-v2 before bad pairs have been removed (same as tree-hr-{given_surname}-similar.csv.gz)\n  - f\"s3://familysearch-names/processed/tree-hr-{given_surname}-similar-v2.csv.gz\"\n- std-buckets - original name buckets\n  - f\"../references/std\\_{given_surname}.txt\"\n- subword-tokenizer - tokenize names into subwords\n  - f\"../data/models/fs-{given_surname}-subword-tokenizer-2048.json\"\n- super-clusters - sets of clusters that were in the same bucket\n  - f\"../data/processed/super*clusters*{given_surname}-{scorer}-{linkage}-{similarity_threshold}-{cluster_freq_normalizer}.json\"\n- swivel_vocab - swivel model vocabulary\n  - f\"../data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n- swivel_model - swivel model\n  - f\"../data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n- test - tree-record frequency (test set, which has rarer names)\n  - f\"../data/processed/tree-hr-{given_surname}-test.csv.gz\"\n- train - tree-record frequency\n  - f\"../data/processed/tree-hr-{given_surname}-train.csv.gz\"\n- train_augments - pairs that were added to the training data\n  - f\"../data/processed/tree-hr-{given_surname}-train-augments.csv.gz\"\n- train_augmented - tree-record frequency augmented with pairs based upon same code and levenshtein similarity\n  - f\"../data/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n\n* tree-hr-names - names from tree-record attachments\n  - f\"s3://fs-nama-data/2024/familysearch-names/interim/tree-hr-{given_surname}/\"\n\n- tree-hr-pairs - pairs of names from tree-record attachments\n  - f\"s3://familysearch-names/interim/tree-hr-{given_surname}-pairs/\"\n- tree-hr-parquet - local copy of tree-hr-names\n  - f\"../data/tree-hr-{given_surname}/\\*.parquet\"\n- tree-hr-parquet-v2 - aggregated pairs of tree-hr with similarity scores\n  - f\"s3://familysearch-names/interim/tree-hr-{given_surname}-aggr-v2.parquet\"\n\n* tree-hr-raw - raw tree-record attachments\n  - f\"s3://fs-nama-data/2024/familysearch-names/raw/tree-hr/\"\n\n## Future work\n\nWe could consider using the swivel output as input to train the bi-encoder (notebook 224) instead of\ncross-encoder-triplets-common-0-augmented. The cross-encoder-triplets-common-0-augmented file has an\nunfortunate \"bump\" in scores at 0.4. That is, a lot of name pairs in the training data are considered\n0.4 similar due to the way the training data was generated. This may make the bi-encoder less-accurate\nthan it could be if it were trained instead with the scores from the swivel model. (Estimate 1 week.)\n\nWe could consider using the swivel output to re-train the original weighted-edit-distance classifier. (Estimate 1 month.)\n\n## Archive\n\n- 10_clean - clean the raw name pairs from FamilySearch (pairs of tree \u003c-\u003e record name) and separate into given and surnames\n  - input: tree-hr-raw\n  - output: tree-hr-names\n- 11_clean_preferred - generate given and surname preferred names from FamilySearch\n  - input: pref-names-raw\n  - output: pref-names-interim\n- 20_generate_pairs - generate pairs from best-matching name pieces\n  - input: tree-hr-names\n  - output: tree-hr-pairs\n- 30_aggregate - aggregate pairs of matching tree \u003c-\u003e record name pieces and compute counts, probabilities, and similarities\n  - input: tree-hr-pairs\n  - output: tree-hr-parquet-v2\n- 31_aggregate_preferred - aggregate preferred names\n  - input: pref-names-interim\n  - output: pref-names\n- 40_filter - convert the tree-hr-attach parquet files into similar and dissimilar name pairs\n  - input: tree-hr-parquet-v2\n  - output: similar-v2, dissimilar-v2\n- 100_train_test_split - split similar names into train and test sets, removing bad pairs\n  - input: similar-v2, pref-names, bad-pairs\n  - output: train-v2, test-v2\n- 200_generate_triplets - generate triplets from training data\n  - input: train-v2\n  - output: triplets\n- 204_generate_subword_tokenizer - train a subword tokenizer\n  - input: triplets, pref-names, train-v2\n  - output: subword-tokenizer\n- 205_generate_common_non_negatives - generate pairs of names that are not negative examples\n  - input: std-buckets, pref-names, triplets, given-nicknames\n  - output: common-non-negatives\n- 206_analyze_triplets - review triplets (optional)\n  - input: triplets, pref-names, common-non-negatives,\n- 207_augment_triplets - augment triplets with additional triplets\n  - input: triplets, pref-names, common-non-negatives, subword-tokenizer\n  - output: triplets-augmented\n- 220_create_language_model_dataset - create large datasets to train roberta masked language model\n  - input: pref-names, tree-hr-parquet(-v2)?\n  - output: all-tree-pref-names, all-tree-hr-names\n- 221_train_language_model - train a roberta masked language model in preparation for training the name-pair cross-encoder\n  - input: all-tree-hr-names-sample, pref-names\n  - output: roberta\n- 222_train_cross_encoder - train a cross-encoder model (based on sentence bert) that takes a pair of names and outputs a similarity score\n  - input: roberta, triplets-augmented\n  - output: cross-encoder\n- 223_generate_triplets_from_cross_encoder - generate various datasets of triplets for training the bi-encoder from the cross-encoder, cause the bi-encoder needs a lot of data\n  - input: pref-names, train-v2, common-non-negatives, std-buckets, cross-encoder\n  - output: cross-encoder-triplets-0 and cross-encoder-triplets-common (run twice)\n- 224_train_bi_encoder - train a bi-encoder model\n  - input: cross-encoder-triplets-common-0-augmented, subword-tokenizer\n  - output: bi-encoder\n- 230_eval_bi_encoder - evaluate a bi-encoder model, used to pick hyperparameters\n  - input: subword-tokenizer, bi-encoder, pref-names, triplets, common-non-negatives\n- 240_create_clusters_from_buckets - split buckets into clusters using the cross encoder; clusters in the same bucket form a super-cluster\n  - input: std-buckets, subword-tokenizer, cross-encoder, bi-encoder, pref-names\n  - output: clusters, super-clusters\n- 241_augment_clusters - augment clusters with additional names that were not in any cluster\n  - input: clusters, subword-tokenizer, pref-names, cross-encoder, bi-encoder\n  - output: augmented-clusters\n- 242_nearby_clusters - compute nearby clusters for each cluster using the bi-encoder followed by the cross-encoder (deprecated)\n  - input: augmented-clusters, subword-tokenizer, bi-encoder, cross-encoder\n  - output: nearby-clusters\n- 243_compress_clusters - compress cluster and super-cluster files so we can check them into git\n  - input: augmented-clusters, super-clusters\n  - output: compressed versions of augmented-clusters and super-clusters\n- 245_eval_coder - compare the precision and recall of nama to familysearch and other coders\n  - input: augmented-clusters, super-clusters, subword-tokenizer, bi-encoder, train-v2, test-v2, query-names, pref-names, given-nicknames\n- 250_create_phonebook - create the phonebook for surnames\n  - input: augmented-clusters, super-clusters, pref-names or hr-names\n  - output: phonebook\n- 251_save_bi_encoder_weights - save the bi-encoder weights so we can use them in fs-nama (java)\n  - input: subword-tokenizer, bi-encoder\n  - output: bi-encoder-weights\n\n## Older Archive\n\n**The information in this system describes a previous version of nama.**\n\nThe previous version has been superceded by the current version described above, but the information below may still be useful.\n\n### Notebooks\n\nRun notebooks in the order listed\n\n- 00_snorkel - Experiment with snorkel (ignore, we tried using snorkel but it didn't work well)\n- 00_snorkel_names - Use snorkel to generate training data (ignore this as well)\n- 10_clean - Clean the raw name pairs from FamilySearch (pairs of tree \u003c-\u003e record name) and separate into given and surnames\n  - input:\n    - hr_raw: \"s3://familysearch-names/raw/tree-hr/\"\n  - output:\n    - hr_names[given]: \"s3://familysearch-names/interim/tree-hr-given/\"\n    - hr_names[surname]: \"s3://familysearch-names/interim/tree-hr-surname/\"\n- 11_clean - Clean the preferred tree names from FamilySearch\n  - input:\n    - pref: \"s3://familysearch-names/raw/tree-preferred/\"\n  - output:\n    - pref_given: \"s3://familysearch-names/interim/tree-preferred-given/\"\n    - pref_surname: \"s3://familysearch-names/interim/tree-preferred-surname/\"\n- 12_analyze_preferred - Review given name abbreviations (optional)\n  - input:\n    - pref_given: \"s3://familysearch-names/interim/tree-preferred-given/\"\n- 20_generate_pairs - Generate pairs of best-matching name pieces from multi-word given or surnames\n  - input:\n    - hr_names: \"s3://familysearch-names/interim/tree-hr-{given_surname}/\"\n  - output:\n    - hr_pairs: \"s3://familysearch-names/interim/tree-hr-{given_surname}-pairs/\"\n- 30_aggregate - Aggregate pairs of matching tree \u003c-\u003e record name pieces and compute counts, probabilities, and similarities\n  - input:\n    - hr_pairs: \"s3://familysearch-names/interim/tree-hr-{given_surname}-pairs/\"\n  - output:\n    - hr_aggr: \"s3://familysearch-names/interim/tree-hr-{given_surname}-aggr.parquet\"\n- 31_aggregate_preferred - Aggregate preferred (tree) names and compute counts\n  - input:\n    - pref: \"s3://familysearch-names/interim/tree-preferred-{given_surname}/\"\n  - output:\n    - pref_aggr: \"s3://familysearch-names/interim/tree-preferred-{given_surname}-aggr.csv.gz\"\n- 32_analyze_aggregate_preferred - Get an idea of how much mass is in the top Nk names (optional)\n  - input:\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- 40_filter - Create lists of pairs that are similarly-spelled (used to create the model) and pairs that are dissimilar\n  - input:\n    - hr_aggr: \"s3://familysearch-names/interim/tree-hr-{given_surname}-aggr.parquet\"\n  - output:\n    - hr_similar: \"s3://familysearch-names/processed/tree-hr-{given_surname}-similar.csv.gz\"\n    - hr_dissimilar: \"s3://familysearch-names/processed/tree-hr-{given_surname}-dissimilar.csv.gz\"\n- 41_generate_nicknames - Create a file of dissimilar given name (potential nickname) pairs for human review\n  - input:\n    - hr_dissimilar_given: \"s3://familysearch-names/processed/tree-hr-given-dissimilar.csv.gz\"\n  - output:\n    - hr_possible_nicknames: \"s3://familysearch-names/processed/tree-hr-nicknames.csv.gz\"\n- 45_train_test_split - Split similar name pairs into training and test sets\n  - input:\n    - hr_similar: \"s3://familysearch-names/processed/tree-hr-{given_surname}-similar.csv.gz\"\n  - output:\n    - hr_train_unfiltered: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-unfiltered.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n- 46_demo_dataset - Create a very small demo dataset to play with (optional)\n  - input:\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - hr_train_unfiltered: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-unfiltered.csv.gz\"\n  - output:\n    - hr_demo_output: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-demo.csv.gz\"\n- 47_bad_pair_generator - Generate hard (near) negative pairs (ignore - not used)\n- 47a_filter_bad_pairs - Remove pairs that were identified as bad pairs during a manual review of borderline pairs\n  - input:\n    - hr_train_unfiltered: s3://familysearch-names/processed/tree-hr-{given_surname}-train-unfiltered.csv.gz\"\n    - bad_pairs: \"s3://familysearch-names/interim/{given_surname}\\_variants_clorinda_reviewed.tsv\"\n  - output:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n- 48_weighted_actual_names_to_csv - Write weighted-actual name pairs for human review (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n  - output:\n    - hr_train_weighted_actuals: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-{size}-weighted-actuals.csv\"\n    - hr_test_weighted_actuals: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test-weighted-actuals.csv\"\n- 49_augment_dataset - Augment the training dataset with other matching pairs based upon names having the same code or levenshtein similarity\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n  - output:\n    - hr_train_augments: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augments.csv.gz\"\n    - hr_train_augmented: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n- 50_autoencoder - Generate an autoencoder model based upon similar name pairs (not used)\n- 51_autoencoder_triplet - Generate a triplet-loss model based upon the autoencoder and near-negatives (not used)\n- 52_glove - Generate a glove model (not used)\n- 60_swivel_tune - Run hyperparameter tuning on swivel model (optional)\n  - input:\n    - hr_train_augmented: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n- 61_swivel - Train a swivel model (takes a long, long, long time)\n  - input:\n    - hr_train_augmented: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n  - output:\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n- 62_swivel_encoder_tune - Tune a swivel-based encoder model (not used)\n- 63_swivel_encoder - Train a swivel-based encoder model (not used)\n- 64_analyze_scores - Compare swivel and levenshtein scores (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n- 65_tfidf - Train a TfidfVectorizer to filter names sent to levenshtein\n  - input:\n    - hr_train_augmented: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n  - output:\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n- 66_ensemble - Train an ensemble model over swivel + levenshtein\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n      output:\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n- 70_compare_similarity - Compare the ensemble model to levenshtein and other floating-score algorithms (optional)\n  - input:\n    - hr_train_augmented: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train-augmented.csv.gz\"\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n- 71_analyze_embeddings - Visualize swivel vectors in 2d space (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n- 72_analyze_names - Analyze name frequencies and codes (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- 80_cluster_tune - Run hyperparameter tuning on clustering model (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n- 81_cluster - Train a clustering model\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - hr_aggr: \"s3://familysearch-names-private/hr-preferred-{given_surname}-aggr.csv.gz\"\n      - comes from FamilySearch, not nama\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n  - output:\n    - clusters: \"s3://nama-data/data/models/fs-{given_surname}-cluster-names.csv\"\n    - cluster_partitions: \"s3://nama-data/data/models/fs-{given_surname}-cluster-partitions.csv\"\n- 82_cluster_levenshtein - Not sure what this does. Very out of date. (ignore)\n- 90_compare_clusters - Compare our clusters to the clusters formed by various coders (soundex, nysiis, etc.) (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n    - clusters: \"s3://nama-data/data/models/fs-{given_surname}-cluster-names.csv\"\n    - cluster_partitions: \"s3://nama-data/data/models/fs-{given_surname}-cluster-partitions.csv\"\n- 91_compare_clusters_old_algo - Get Statistics for the old clusters (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n- 92_compare_oov_approaches - Compare our approach to handling out of vocab names to four simpler approaches (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - hr_test: \"s3://familysearch-names/processed/tree-hr-{given_surname}-test.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n    - clusters: \"s3://nama-data/data/models/fs-{given_surname}-cluster-names.csv\"\n    - cluster_scores: \"s3://nama-data/data/processed/fs-{given_surname}-cluster-scores-{vocab_size}-{embed_dim}-precomputed.jsonl.gz\"\n      - see 99_precompute\n    - hr_aggr: \"s3://familysearch-names/interim/tree-hr-{given_surname}-aggr.parquet\"\n- 97_analyze_nicknames - Review nicknames (optional)\n  - input:\n    - nicknames: \"../references/givenname_nicknames.csv\"\n- 98_given_surname_freq - Generate how likely a name is to be a given vs surname (optional)\n  - input:\n    - pref_aggr[given]: \"s3://familysearch-names/processed/tree-preferred-given-aggr.csv.gz\"\n    - pref_aggr[surname]: \"s3://familysearch-names/processed/tree-preferred-surname-aggr.csv.gz\"\n  - output:\n    - given_surname_freq: \"s3://familysearch-names/processed/tree-preferred-given-surname-freq.csv.gz\"\n- 99_precompute - Pre-compute embeddings and cluster-scores so they can be cached (optional)\n  - input:\n    - hr_train: \"s3://familysearch-names/processed/tree-hr-{given_surname}-train.csv.gz\"\n    - pref_aggr: \"s3://familysearch-names/processed/tree-preferred-{given_surname}-aggr.csv.gz\"\n    - swivel_vocab: \"s3://nama-data/data/models/fs-{given_surname}-swivel-vocab-{vocab_size}-augmented.csv\"\n    - swivel_model: \"s3://nama-data/data/models/fs-{given_surname}-swivel-model-{vocab_size}-{embed_dim}-augmented.pth\"\n    - tfidf_model: \"s3://nama-data/data/models/fs-{given_surname}-tfidf.joblib\"\n    - ensemble_model: \"s3://nama-data/data/models/fs-{given_surname}-ensemble-model-{vocab_size}-{embed_dim}-augmented-{negative_multiplier}.joblib\"\n    - clusters: \"s3://nama-data/data/models/fs-{given_surname}-cluster-names.csv\"\n  - output:\n    - cluster_scores: \"s3://nama-data/data/processed/fs-{given_surname}-cluster-scores-{vocab_size}-{embed_dim}-precomputed.jsonl.gz\"\n    - embeddings: \"s3://nama-data/data/processed/fs-{given_surname}-embeddings-{vocab_size}-{embed_dim}-precomputed.jsonl.gz\"\n    - cluster_embeddings: \"s3://nama-data/data/processed/fs-{given_surname}-cluster-embeddings-{vocab_size}-{embed_dim}-precomputed.jsonl.gz\"\n\n### Server\n\nThe server is currently a out of date.\n\n#### Starting the server and online server documentation\n\n    uvicorn src.server.server:app --reload\n    http://localhost:8000/docs\n\n#### Using docker\n\n    docker build -t nama .\n    docker run --rm -d --name nama -p 8080:8080 nama\n\n### Using Fastec2 for managing remote jupyter notebooks\n\n- Instructions: https://www.fast.ai/2019/02/15/fastec2/\n- pip install git+https://github.com/fastai/fastec2.git\n- ssh-add default.pem\n- fe2 launch \u003c name \u003e base 80 r5.2xlarge # 80Gb disk, 64Gb memory\n- ./remote-install.sh \u003c ip \u003e\n- fe2 connect \u003c name \u003e 8888\n  - cd nama\n  - conda activate nama\n  - jupyter notebook\n- fe2 stop \u003c name \u003e # stopped instances can be re-started with fe2 start \u003c name \u003e\n- fe2 terminate \u003c name \u003e # releases instance name and associated disk\n\n### Using gcloud\n\n- Use their deeplearning ami, and don't try to install another version of pytorch\n- don't use conda\n\n### Logging into Weights and Biases\n\n- run `wandb login`\n- login information will be added to your ~/.netrc file\n- copy just that information from ~/.netrc to ~/.netrc.wandb. This file will be added to machines launched with FastEC2\n\n## Data\n\n### locations\n\n- raw data can be found at s3://familysearch-names/raw\n- large interim data files can be found at s3://familysearch-names/interim\n\n### descriptions\n\n- tree_hr_given-similar.csv + tree-hr-surname-similar.csv - likely-similar name pairs\n  - name - preferred name in the tree\n  - alt_name - name in the record (pairs are omitted when alt_name is the same as name)\n  - frequency - how many times these two names appear together\n  - reverse_frequency - how many times the alt_name is the name in the tree and the name is the name in the record\n  - sum_name_frequency - how many times the (tree) name appears as a (tree) name\n  - total_name_frequency - how many times the (tree) name appears as a (tree) name or as an alt_name\n  - total_alt_name_frequency - how many times the alt_name appears as an alt_name or a (tree) name\n  - ordered_prob - frequency / sum_name_frequency\n  - unordered_prob - (frequency + reverse_frequency) / the total number of times either name appears as a name or an alt_name\n  - similarity - 1.0 - (levenshtein distance between the name and alt_name / max(len(name), len)alt_name)))\n\n## Project Organization\n\n    ├── LICENSE\n    ├── Makefile           \u003c- Makefile with commands like `make data` or `make train`\n    ├── README.md          \u003c- The top-level README for developers using this project.\n    ├── data\n    │\n    ├── docs               \u003c- A default Sphinx project; see sphinx-doc.org for details\n    │\n    ├── models             \u003c- Trained and serialized models, model predictions, or model summaries\n    │\n    ├── notebooks          \u003c- Jupyter notebooks. Naming convention is a number (for ordering),\n    │                         the creator's initials, and a short `-` delimited description, e.g.\n    │                         `1.0-jqp-initial-data-exploration`.\n    │\n    ├── references         \u003c- Data dictionaries, manuals, and all other explanatory materials.\n    │\n    ├── reports            \u003c- Generated analysis as HTML, PDF, LaTeX, etc.\n    │   └── figures        \u003c- Generated graphics and figures to be used in reporting\n    │\n    ├── requirements.txt   \u003c- The requirements file for reproducing the analysis environment, e.g.\n    │                         generated with `pip freeze \u003e requirements.txt`\n    │\n    ├── setup.py           \u003c- makes project pip installable (pip install -e .) so src can be imported\n    ├── src                \u003c- Source code for use in this project.\n    │   ├── __init__.py    \u003c- Makes src a Python module\n    │   ├── data           \u003c- Functions to download or generate data\n    │   ├── eval           \u003c- Functions to evaluate models\n    │   ├── features       \u003c- Functions to turn raw data into features for modeling\n    │   ├── models         \u003c- Functions to train models and then use trained models to make predictions\n    │   ├── server         \u003c- Simple server\n    │   └── visualization  \u003c- empty for now\n    ├── tests              \u003c- Tests for source code\n    │\n    └── tox.ini            \u003c- tox file with settings for running tox; see tox.readthedocs.io\n\n---\n\n\u003cp\u003e\u003csmall\u003eProject based on the \u003ca target=\"_blank\" href=\"https://drivendata.github.io/cookiecutter-data-science/\"\u003ecookiecutter data science project template\u003c/a\u003e. #cookiecutterdatascience\u003c/small\u003e\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frootsdev%2Fnama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frootsdev%2Fnama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frootsdev%2Fnama/lists"}