### Create and Activate Conda Environment Source: https://github.com/thompsonb/vecalign/blob/master/README.md Sets up a Python 3.7 environment named 'vecalign' using conda, installing necessary packages like cython and numpy. Ensure conda is updated and the environment is activated before proceeding. ```bash # Use latest conda conda update conda -y # Create conda environment conda create --force -y --name vecalign python=3.7 # Activate new environment source `conda info --base`/etc/profile.d/conda.sh # See: https://github.com/conda/conda/issues/7980 conda activate vecalign # Install required packages conda install -y -c anaconda cython conda install -y -c anaconda numpy pip install mcerp ``` -------------------------------- ### Vecalign Output Format Source: https://github.com/thompsonb/vecalign/blob/master/README.md Example of Vecalign's standard output, detailing sentence alignment. Each line represents an alignment with source and target sentence indices, followed by the computed alignment cost. Costs are normalized but do not include penalties for multiple sentences per alignment; insertions/deletions have a cost of zero. ```text [0]:[0]:0.156006 [1]:[1]:0.160997 [2]:[2]:0.217155 [3]:[3]:0.361439 [4]:[4]:0.346332 [5]:[5]:0.211873 [6]:[6, 7, 8]:0.507506 [7]:[9]:0.252747 [8, 9]:[10, 11, 12]:0.139594 [10, 11]:[13]:0.273751 [12]:[14]:0.165397 [13]:[15, 16, 17]:0.436312 [14]:[18, 19, 20, 21]:0.734142 []:[22]:0.000000 []:[23]:0.000000 []:[24]:0.000000 []:[25]:0.000000 [15]:[26, 27, 28]:0.840094 ... ``` -------------------------------- ### Vecalign CLI Parameters Reference Source: https://context7.com/thompsonb/vecalign/llms.txt Command-line parameters for the vecalign.py script. Includes required parameters for source/target files and embeddings, and optional parameters for gold alignments, alignment constraints, and debugging. ```bash ./vecalign.py \ # Required parameters --src file1.src file2.src # Source files to align --tgt file1.tgt file2.tgt # Target files to align --src_embed overlaps.txt overlaps.emb # Source embeddings (text + binary) --tgt_embed overlaps.txt overlaps.emb # Target embeddings (text + binary) # Optional parameters --gold gold1.txt gold2.txt # Gold alignments for scoring --alignment_max_size 4 # Max N+M for N-M alignments (default: 4) --one_to_many [50] # One-to-many mode with optional M limit --del_percentile_frac 0.2 # Deletion penalty percentile (default: 0.2) --max_size_full_dp 300 # Threshold for full DP (default: 300) --costs_sample_size 20000 # Cost estimation samples (default: 20000) --num_samps_for_norm 100 # Normalization samples (default: 100) --search_buffer_size 5 # Search buffer width (default: 5) --print_aligned_text # Print aligned text for debugging --debug_save_stack stack.pkl # Save internal state to pickle --verbose # Enable detailed logging ``` -------------------------------- ### Align and Score Against Gold Reference Alignments Source: https://context7.com/thompsonb/vecalign/llms.txt This command aligns multiple file pairs and evaluates them against gold reference alignments, calculating strict and lax precision, recall, and F1 scores. Output is directed to /dev/null, but scores are printed to stderr. ```bash # Align and score against gold reference alignments ./vecalign.py --alignment_max_size 8 \ --src bleualign_data/test*.de \ --tgt bleualign_data/test*.fr \ --gold bleualign_data/test*.defr \ --src_embed bleualign_data/overlaps.de bleualign_data/overlaps.de.emb \ --tgt_embed bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb > /dev/null # Output to stderr: # --------------------------------- # | | Strict | Lax | # | Precision | 0.899 | 0.985 | # | Recall | 0.904 | 0.987 | # | F1 | 0.902 | 0.986 | # --------------------------------- ``` -------------------------------- ### Evaluate Alignment Quality with Standalone Tool Source: https://context7.com/thompsonb/vecalign/llms.txt Use this tool to evaluate alignment quality by comparing test alignments against gold reference alignments without running the alignment algorithm. It supports single and multiple file pair evaluations. ```bash # Score test alignments against gold reference ./score.py -t test_alignments.txt -g gold_alignments.txt # Multiple file pairs ./score.py -t test1.txt test2.txt test3.txt \ -g gold1.txt gold2.txt gold3.txt # Alignment file format (one alignment per line): # [src_indices]:[tgt_indices] # Example: # [0]:[0] # [1]:[1, 2] # [2, 3]:[3] ``` -------------------------------- ### Run Vecalign with Gold Alignment Scoring Source: https://github.com/thompsonb/vecalign/blob/master/README.md Aligns sentences and scores the results against a gold standard alignment using the '-g' flag. Supports multiple source, target, and gold files using wildcards. Output is redirected to /dev/null, focusing on the scoring metrics. ```bash ./vecalign.py --alignment_max_size 8 --src bleualign_data/test*.de --tgt bleualign_data/test*.fr \ --gold bleualign_data/test*.defr \ --src_embed bleualign_data/overlaps.de bleualign_data/overlaps.de.emb \ --tgt_embed bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb > /dev/null ``` -------------------------------- ### Embed Documents using LASER Source: https://github.com/thompsonb/vecalign/blob/master/README.md Embed the generated overlap files using the LASER toolkit. Ensure the LASER environment variable is set and specify the language code for the embeddings. ```bash $LASER/tasks/embed/embed.sh bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb [fra] ``` ```bash $LASER/tasks/embed/embed.sh bleualign_data/overlaps.de bleualign_data/overlaps.de.emb [deu] ``` -------------------------------- ### Perform Basic Sentence Alignment with Pre-computed Embeddings Source: https://context7.com/thompsonb/vecalign/llms.txt Use this command to perform basic sentence alignment between source and target documents using pre-computed sentence embeddings. The output format indicates source and target sentence indices and the alignment cost. ```bash # Basic sentence alignment with pre-computed embeddings ./vecalign.py --alignment_max_size 8 \ --src bleualign_data/dev.de \ --tgt bleualign_data/dev.fr \ --src_embed bleualign_data/overlaps.de bleualign_data/overlaps.de.emb \ --tgt_embed bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb # Output format: [src_indices]:[tgt_indices]:alignment_cost # [0]:[0]:0.156006 # [1]:[1]:0.160997 # [6]:[6, 7, 8]:0.507506 <- 1-to-3 alignment # [8, 9]:[10, 11, 12]:0.139594 <- 2-to-3 alignment # []:[22]:0.000000 <- deletion (no source match) ``` -------------------------------- ### Run Vecalign for Sentence Alignment Source: https://github.com/thompsonb/vecalign/blob/master/README.md Executes Vecalign to align sentences between source and target files using provided embeddings. Alignments are output to stdout, showing sentence indices and alignment costs. The --alignment_max_size flag controls the maximum number of sentences to group. ```bash ./vecalign.py --alignment_max_size 8 --src bleualign_data/dev.de --tgt bleualign_data/dev.fr \ --src_embed bleualign_data/overlaps.de bleualign_data/overlaps.de.emb \ --tgt_embed bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb ``` -------------------------------- ### Generate Overlap Files with Vecalign Source: https://github.com/thompsonb/vecalign/blob/master/README.md Use the overlap.py script to generate overlap files for French and German documents. The -n option specifies the number of sentences to consider for combinations. ```bash ./overlap.py -i bleualign_data/dev.fr bleualign_data/test*.fr -o bleualign_data/overlaps.fr -n 10 ``` ```bash ./overlap.py -i bleualign_data/dev.de bleualign_data/test*.de -o bleualign_data/overlaps.de -n 10 ``` -------------------------------- ### Build Document Embedding with Sentence Vectors Source: https://context7.com/thompsonb/vecalign/llms.txt Generates a document embedding that preserves sentence order information. Requires sentence vectors and their frequency counts. Output shape is (sent_emb_size * NUM_TIME_SLOTS,). Document similarity can be calculated using cosine similarity. ```python import numpy as np from standalone_document_embedding_demo import build_doc_embedding # Sentence embeddings (from LASER, LaBSE, etc.) # Recommended: project to lower dimension (e.g., 32) using PCA first sent_emb_size = 32 n_sents = 100 sent_vecs = np.random.rand(n_sents, sent_emb_size) - 0.5 # Replace with real embeddings # Sentence frequency counts from corpus (for IDF-like weighting) # Higher counts = lower weight sent_counts = np.array([1, 5, 2, 10, 3, ...]) # Count per sentence # Build document embedding # Uses PERT distributions to create position-aware time slots doc_emb = build_doc_embedding(sent_vecs, sent_counts) # Output shape: (sent_emb_size * NUM_TIME_SLOTS,) = (32 * 16,) = (512,) # Document similarity via cosine: np.dot(doc_emb1, doc_emb2) print(f'Document Embedding Shape: {doc_emb.shape}') ``` -------------------------------- ### Read Embeddings and Create Document Matrices Source: https://context7.com/thompsonb/vecalign/llms.txt These functions handle reading pre-computed sentence embeddings and constructing document embedding matrices. The `make_doc_embedding` function supports many-to-many alignments by including embeddings for overlapping sentence sequences. ```python from dp_utils import read_in_embeddings, make_doc_embedding # Read embeddings from text file (sentences) and binary file (embeddings) # Text file: one sentence per line # Binary file: float32 array, shape (num_sentences, embedding_dim) sent2line, line_embeddings = read_in_embeddings( 'overlaps.txt', # Text file with overlap sentences 'overlaps.emb' # Binary embedding file (float32) ) # sent2line: dict mapping sentence text -> embedding index # line_embeddings: numpy array shape (num_sentences, 1024) # Create document embedding matrix for alignment # Includes embeddings for sentence overlaps (for many-to-many alignments) doc_lines = open('document.txt', 'rt', encoding='utf-8').readlines() num_overlaps = 4 # Support up to 4 consecutive sentence alignments doc_vecs = make_doc_embedding( sent2line, line_embeddings, doc_lines, num_overlaps ) # Returns: numpy array shape (num_overlaps, num_sentences, embedding_dim) # doc_vecs[0] = single sentence embeddings # doc_vecs[1] = consecutive pair embeddings # doc_vecs[2] = consecutive triple embeddings # doc_vecs[3] = consecutive quadruple embeddings ``` -------------------------------- ### Generate Sentence Overlap Files Source: https://context7.com/thompsonb/vecalign/llms.txt This command generates text files containing all sentence overlaps (concatenations of consecutive sentences) required for embedding. This is a preprocessing step for Vecalign to handle many-to-many alignments. ```bash # Generate overlap file with up to 10 consecutive sentence combinations ./overlap.py -i bleualign_data/dev.fr bleualign_data/test*.fr \ -o bleualign_data/overlaps.fr \ -n 10 ./overlap.py -i bleualign_data/dev.de bleualign_data/test*.de \ -o bleualign_data/overlaps.de \ -n 10 # The output files contain: # - Single sentences (overlap=1) # - Pairs of consecutive sentences (overlap=2) # - Triples of consecutive sentences (overlap=3) # - ... up to n consecutive sentences ``` -------------------------------- ### Score Alignments Against Gold References Source: https://context7.com/thompsonb/vecalign/llms.txt Calculate precision, recall, and F1 scores for test alignments by comparing them against gold standard references. This involves reading alignment files and using the `score_multiple` function. The `value_for_div_by_0` parameter handles potential division by zero errors. ```python from dp_utils import read_alignments from score import score_multiple, log_final_scores # Read alignment files # Format: [src_indices]:[tgt_indices] per line gold_alignments = [read_alignments('gold1.txt'), read_alignments('gold2.txt')] test_alignments = [read_alignments('test1.txt'), read_alignments('test2.txt')] # Compute strict and lax precision/recall/F1 results = score_multiple( gold_list=gold_alignments, test_list=test_alignments, value_for_div_by_0=0.0 # Value when division by zero occurs ) # results dict contains: # - precision_strict, precision_lax # - recall_strict, recall_lax # - f1_strict, f1_lax print(f"Strict F1: {results['f1_strict']:.3f}") print(f"Lax F1: {results['f1_lax']:.3f}") # Pretty-print results table log_final_scores(results) ``` -------------------------------- ### Vecalign Paper Citation Source: https://github.com/thompsonb/vecalign/blob/master/README.md Citation for the Vecalign paper: 'Vecalign: Improved Sentence Alignment in Linear Time and Space'. ```bibtex @inproceedings{thompson-koehn-2019-vecalign, title = "{V}ecalign: Improved Sentence Alignment in Linear Time and Space", author = "Thompson, Brian and Koehn, Philipp", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)", month = nov, year = "2019", address = "Hong Kong, China", publisher = "Association for Computational Linguistics", url = "https://www.aclweb.org/anthology/D19-1136", doi = "10.18653/v1/D19-1136", pages = "1342--1348", } ``` -------------------------------- ### Embed Overlapping Sentences with LASER Source: https://context7.com/thompsonb/vecalign/llms.txt Embed generated overlap files using a multilingual sentence encoder like LASER. Embeddings must be stored as binary float32 files. Ensure the LASER environment variable is set correctly. Old embeddings should be deleted before re-embedding. ```bash # Set LASER environment variable to your LASER installation path export LASER=/path/to/LASER # Embed French overlaps $LASER/tasks/embed/embed.sh bleualign_data/overlaps.fr bleualign_data/overlaps.fr.emb [fra] # Embed German overlaps $LASER/tasks/embed/embed.sh bleualign_data/overlaps.de bleualign_data/overlaps.de.emb [deu] # Note: LASER will not overwrite existing files - delete old embeddings first: rm bleualign_data/overlaps.fr.emb bleualign_data/overlaps.de.emb ``` -------------------------------- ### Perform Core Sentence Alignment Source: https://context7.com/thompsonb/vecalign/llms.txt Use this function to perform sentence alignment on pre-processed embedding matrices. It requires loading embeddings, creating document embedding matrices, and defining alignment types. Adjust parameters like `del_percentile_frac` and `width_over2` for fine-tuning. ```python from dp_utils import vecalign, make_alignment_types, read_in_embeddings, make_doc_embedding # Load pre-computed embeddings src_sent2line, src_embeddings = read_in_embeddings('overlaps.src', 'overlaps.src.emb') tgt_sent2line, tgt_embeddings = read_in_embeddings('overlaps.tgt', 'overlaps.tgt.emb') # Read source and target documents src_lines = open('document.src', 'rt', encoding='utf-8').readlines() tgt_lines = open('document.tgt', 'rt', encoding='utf-8').readlines() # Create document embedding matrices (with overlaps for many-to-many alignments) alignment_max_size = 8 vecs0 = make_doc_embedding(src_sent2line, src_embeddings, src_lines, alignment_max_size) vecs1 = make_doc_embedding(tgt_sent2line, tgt_embeddings, tgt_lines, alignment_max_size) # Define alignment types to consider alignment_types = make_alignment_types(alignment_max_size) # Returns: [(1,1), (1,2), (1,3), (2,1), (2,2), (3,1), ...] where n+m <= max_size # Run alignment algorithm stack = vecalign( vecs0=vecs0, vecs1=vecs1, final_alignment_types=alignment_types, del_percentile_frac=0.2, # Deletion penalty percentile width_over2=5, # Search buffer width max_size_full_dp=300, # Threshold for full vs sparse DP costs_sample_size=20000, # Samples for cost estimation num_samps_for_norm=100 # Samples for normalization ) # Access results alignments = stack[0]['final_alignments'] scores = stack[0]['alignment_scores'] for (src_idxs, tgt_idxs), score in zip(alignments, scores): print(f'{src_idxs}:{tgt_idxs}:{score:.6f}') ``` -------------------------------- ### Vecalign Scoring Results Source: https://github.com/thompsonb/vecalign/blob/master/README.md Presents typical scoring results (Precision, Recall, F1) for Vecalign when compared against a gold alignment. These metrics indicate the accuracy of the sentence alignments produced by the algorithm. ```text --------------------------------- | | Strict | Lax | | Precision | 0.899 | 0.985 | | Recall | 0.904 | 0.987 | | F1 | 0.902 | 0.986 | --------------------------------- ``` -------------------------------- ### Document Alignment Paper Citation Source: https://github.com/thompsonb/vecalign/blob/master/README.md Citation for the document alignment paper: 'Exploiting Sentence Order in Document Alignment'. ```bibtex @inproceedings{thompson-koehn-2020-exploiting, title = "Exploiting Sentence Order in Document Alignment", author = "Thompson, Brian and Koehn, Philipp", booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)", month = nov, year = "2020", address = "Online", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2020.emnlp-main.483", doi = "10.18653/v1/2020.emnlp-main.483", pages = "5997--6007", } ``` -------------------------------- ### Perform One-to-Many Sentence Alignment Source: https://context7.com/thompsonb/vecalign/llms.txt Use this specialized alignment mode to restrict the source side to single sentences while allowing multiple target sentences per alignment. This is useful for aligning documents where source and target segmentation differs. ```bash # One-to-many alignment (1:1, 1:2, ..., 1:M) # Default M=50 when flag is used without argument ./vecalign.py --one_to_many 50 \ --src document_src.txt \ --tgt document_tgt.txt \ --src_embed overlaps.src overlaps.src.emb \ --tgt_embed overlaps.tgt overlaps.tgt.emb # Useful for aligning documents where source has been segmented differently than target ``` -------------------------------- ### Generate Sentence Overlaps for Embeddings Source: https://context7.com/thompsonb/vecalign/llms.txt This function generates sequences of overlapping sentences from a document, which are used for creating embedding matrices. It supports generating overlaps up to a specified number of consecutive sentences. The output can be written to a file for subsequent embedding processing. ```python from dp_utils import yield_overlaps # Read document sentences lines = open('document.txt', 'rt', encoding='utf-8').readlines() # Generate all overlaps up to 10 consecutive sentences num_overlaps = 10 overlap_sentences = set() for overlap_line in yield_overlaps(lines, num_overlaps): overlap_sentences.add(overlap_line) # Write unique overlaps to file for embedding with open('overlaps.txt', 'wt', encoding='utf-8') as f: for line in sorted(overlap_sentences): f.write(line + '\n') # Each line in output is either: # - A single sentence # - Two consecutive sentences joined by space # - Three consecutive sentences joined by space # ... up to num_overlaps consecutive sentences ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.