### Example Clone Pair Output

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

An example of the output format for detected clone pairs, represented as comma-separated file IDs.

```text
1,2
2,3
```

--------------------------------

### Initialize Nodes for Indexing

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Modify `runnodes.sh` to initialize multiple nodes concurrently. Each node requires a unique properties file location. Run `./runnodes.sh [num_nodes]` to start the initialization.

```bash
for i in $(seq 1 1 $num_nodes)
do
    java -Dproperties.location="$rootPATH/NODE_$i/sourcerer-cc.properties" -Xms10g -Xmx10g  -jar dist/indexbased.SearchManager.jar init $threshold &
done
```

--------------------------------

### Index Code Files

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Modify `runnodes.sh` to index code files across multiple nodes. Ensure the `threshold` variable is set appropriately. Run `./runnodes.sh [num_nodes]` to start indexing.

```bash
for i in $(seq 1 1 $num_nodes)
do
    java -Dproperties.location="$rootPATH/NODE_$i/sourcerer-cc.properties" -Xms20g -Xmx20g  -jar dist/indexbased.SearchManager.jar index $threshold &
done
```

--------------------------------

### Prepare Node Directory Structure with execute.sh

Source: https://context7.com/mondego/sourcerercc/llms.txt

Sets up individual node directories, splits query files, and copies configuration. Use for initializing nodes for search, init, or index modes.

```bash
./execute.sh search 10 8
```

```bash
./execute.sh init 1 8
```

--------------------------------

### Sample Project Paths List

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

A sample of the 'paths.txt' file, listing the .zip archives of projects to be processed.

```text
path/for/projects/aesthetic-master.zip
path/for/projects/aesthetic-master.zipOffsetAnimator-master.zip
path/for/projects/aesthetic-master.zipResourceInspector-master.zip
path/for/projects/aesthetic-master.zipzachtaylor-JPokemon.zip
```

--------------------------------

### Running the WebApp with Flask

Source: https://github.com/mondego/sourcerercc/blob/master/requirements.txt

Set environment variables and run the Flask development server. Ensure FLASK_APP is set to your application's entry point and FLASK_ENV is set to development.

```bash
EXPORT FLASK_APP=hello
EXPORT FLASK_ENV=development
flask run
```

--------------------------------

### Running the Tokenizer

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Execute the tokenizer script with the specified project archive extension.

```bash
python tokenizer.py zip
```

--------------------------------

### Tokenizer Input Project List Configuration

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Specify the input file containing a list of project paths to be analyzed by the tokenizer.

```ini
FILE_projects_list = this/is/a/path/paths.txt
```

--------------------------------

### Initialize and Index with SearchManager

Source: https://context7.com/mondego/sourcerercc/llms.txt

Use SearchManager to build inverted and forward Lucene indexes. Run initialization once, then indexing for each node partition. Ensure Java heap space is adequately allocated.

```bash
java -Dproperties.rootDir="$(pwd)/" \
     -Dproperties.location="$(pwd)/NODE_1/sourcerer-cc.properties" \
     -Dlog4j.configurationFile="$(pwd)/NODE_1/log4j2.xml" \
     -Xms10g -Xmx10g \
     -jar dist/indexbased.SearchManager.jar init 8
```

```bash
java -Dproperties.rootDir="$(pwd)/" \
     -Dproperties.location="$(pwd)/NODE_1/sourcerer-cc.properties" \
     -Dlog4j.configurationFile="$(pwd)/NODE_1/log4j2.xml" \
     -Xms20g -Xmx20g \
     -jar dist/indexbased.SearchManager.jar index 8
```

--------------------------------

### Parse Source Files into Token Bags (Bash)

Source: https://context7.com/mondego/sourcerercc/llms.txt

Prepare a projects.txt file, number the lines, initialize the ID generator, and run the parser script. This process tokenizes Java source files and writes token-frequency bags and bookkeeping IDs.

```bash
# Step 1: prepare a projects.txt with lines: <project_id>,<path_to_project_root>
echo "1,/repos/project-alpha" > projects.txt
echo "2,/repos/project-beta"  >> projects.txt

# Step 2: number the lines and initialise the ID generator
nl -ba -s ',' projects.txt > projects_numbered.txt
echo 0 > idgen.txt
echo "f" > idgenstatus.txt

# Step 3: run the parser (3 parallel processes splitting the list)
./runparser.sh 3

# Output: input/dataset/numberedProjects.aa_clone-input.txt  (token bags)
#         input/bookkeeping/numberedProjects.aa_idFile.txt    (ID map)
# Token-bag line format:
#   <project_id>,<file_id>@#@token1@@::@@freq1,token2@@::@@freq2,...
# Example line:
#   1,1001@#@public@@::@@3,void@@::@@2,String@@::@@5,return@@::@@1
```

--------------------------------

### Tokenizer Configuration Parameters

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Configure performance parameters for the tokenizer, including the number of active processes and the batch size of projects each process handles.

```ini
N_PROCESSES = 1
; How many projects does each process process at a time?
PROJECTS_BATCH = 2
```

--------------------------------

### Prepare Input for SourcererCC

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Concatenate tokenized files into a single block file and copy it to the SourcererCC input directory.

```bash
cat files_tokens/* > blocks.file
cp blocks.file SourcererCC/clone-detector/input/dataset/
```

--------------------------------

### Build SourcererCC

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Execute the `ant cdi` command to build the SourcererCC project.

```bash
ant cdi
```

--------------------------------

### Build SourcererCC Project JARs (Bash)

Source: https://context7.com/mondego/sourcerercc/llms.txt

Use Ant to compile Java sources and package the main index-based search manager JAR, no-index clone detector JARs, and the index merger JAR. A clean build option is also available.

```bash
# Compile all sources and create the index-based search JAR
ant cdi

# Create the no-index clone detector JARs (with and without prefix filter)
ant cdboth

# Create the index merger JAR
ant cdmerge

# Full clean build of all targets
ant clean cdi cdboth cdmerge
```

--------------------------------

### Run the full index-based SourcererCC pipeline

Source: https://context7.com/mondego/sourcerercc/llms.txt

Execute the complete SourcererCC workflow for large-scale clone detection. This can be done fully automated via the Python controller or step-by-step using shell commands.

```bash
# ── Option A: Fully automated via Python controller ──────────────────────────
# Prerequisites: projects.txt listing <id>,<path> pairs; ant on PATH; Java 7+
./runparser.sh 3               # parse source files into token bags
python controller.py 10        # init → index → merge → search with 10 nodes
# Clone results: NODE_*/output/0.8/*clones_index_WITH_FILTER.txt

# ── Option B: Step-by-step manual run ────────────────────────────────────────
# 0. Build
ant cdi

# 1. Parse source files (3 parallel parser processes)
./runparser.sh 3

# 2. Prepare 1-node structure for init/index
./execute.sh init 1 8

# 3. Init — build global token-position map
./runnodes.sh init 1 8

# 4. Index — build per-node Lucene indexes
./runnodes.sh index 1 8

# 5. Merge distributed indexes into one
./mergeindexes.sh

# 6. Prepare N-node structure for search
./execute.sh search 10 8

# 7. Search — find clone pairs
./runnodes.sh search 10 8

# Clone output per node:

```

--------------------------------

### Configure SourcererCC Properties (Properties)

Source: https://context7.com/mondego/sourcerercc/llms.txt

Central configuration file for SourcererCC, controlling token bounds, sharding, threading, and I/O paths. Files outside MIN_TOKENS and MAX_TOKENS are ignored. Sharding splits the token space to speed up large-scale searches.

```properties
# sourcerer-cc.properties

# I/O directories
NODE_PREFIX=NODE
QUERY_DIR_PATH=query
OUTPUT_DIR=output
DATASET_DIR_PATH=input/dataset

# Token bounds — files with token counts outside [MIN, MAX] are ignored
MIN_TOKENS=65
MAX_TOKENS=500000

# Sharding: split token space into buckets to speed up large-scale search
# Shard boundaries: 65-65, 66-100, 101-300, 301-500000, 500001-500000
IS_SHARDING=true
SHARD_MAX_NUM_TOKENS=65,100,300,500000

# Thread counts for indexing pipeline
BTSQ_THREADS=4   # bag-to-sort queue
BTIIQ_THREADS=4  # bag-to-inverted-index queue
BTFIQ_THREADS=4  # bag-to-forward-index queue

# Thread counts for search pipeline
QLQ_THREADS=4    # query line queue
QBQ_THREADS=4    # query block queue
QCQ_THREADS=4    # query candidates queue
VCQ_THREADS=16   # verify candidate queue
RCQ_THREADS=4    # report clone queue

# Logging/recovery
IS_STATUS_REPORTER_ON=true
IS_GEN_CANDIDATE_STATISTICS=false
LOG_PROCESSED_LINENUMBER_AFTER_X_LINES=50
```

--------------------------------

### Run SourcererCC Controller

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Execute the main controller script for SourcererCC to initiate the clone detection process.

```python
python controller.py
```

--------------------------------

### Display Tokenizer Configuration Names

Source: https://github.com/mondego/sourcerercc/blob/master/WebApp/templates/upload.html

Iterates through a list of tokenizer configurations to display their names. The first configuration is displayed directly, and the rest are displayed in a loop.

```html
{{ exp_config[0].name }} {% for exp in exp_config[1:] %} {{ exp.name }} {% endfor %}
```

--------------------------------

### Language Primitive and File Extension Configuration

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Configure language-specific primitives for comment handling and specify the file extensions to be analyzed.

```ini
[Language]
comment_inline = //
comment_open_tag = /*
comment_close_tag = */
File_extensions = .py
```

--------------------------------

### Run CloneDetectorWithFilter with different positional filters

Source: https://context7.com/mondego/sourcerercc/llms.txt

Execute CloneDetectorWithFilter with various positional filter strategies and similarity metrics. The `run.sh` script automates running all combinations.

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetectorWithFilter.jar myproject 8 0 pos_filter_none overlap
```

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetectorWithFilter.jar myproject 8 0 pos_filter_cv overlap
```

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetectorWithFilter.jar myproject 8 0 pos_filter_c jaccard
```

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetectorWithFilter.jar myproject 8 0 pos_filter_v jaccard
```

```bash
./run.sh 1 myproject 8
```

--------------------------------

### Configure Clone Detection Bounds

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Set the minimum and maximum token counts for files to be considered for clone detection. These parameters are defined in the sourcerer-cc.properties file.

```properties
# Ignore all files outside these bounds
MIN_TOKENS=65
MAX_TOKENS=500000
```

--------------------------------

### Configure Search Properties

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Set `min_tokens` and `max_tokens` in `sourcerer-cc.properties` to define the token range for search. Also, set `SEARCH_SHARD_ID` to the appropriate shard number.

```properties
SEARCH_SHARD_ID=<shrad id>
shard id is 1 for 65-75, 2 for 76-90, and so on. (Yes it is manual right now)
```

--------------------------------

### Configure Indexing Properties

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Set these properties in `sourcerer-cc.properties` to control indexing behavior, including sharding and token limits.

```properties
IS_SHARDING=true
MIN_TOKENS=65
MAX_TOKENS=500000
SHARD_MAX_NUM_TOKENS=<comma seperated list of numbers. 75,90,100  would mean we want 4 shards: 65-75, 76-90, 91-100, 101-500000>
```

--------------------------------

### Merge Index Files

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Run the `IndexMerger.jar` to merge the indexed data. This command requires the main `sourcerer-cc.properties` file and significant heap memory allocation.

```java
java -Dproperties.location="$rootPATH/sourcerer-cc.properties" -Xms20g -Xmx20g  -jar dist/indexbased.IndexMerger.jar merge
```

--------------------------------

### Run Multiple Nodes in Parallel with runnodes.sh

Source: https://context7.com/mondego/sourcerercc/llms.txt

Orchestrates multiple SearchManager processes in parallel for index or search modes. Monitors node success and reports overall exit status.

```bash
./runnodes.sh index 50 8
```

```bash
./runnodes.sh search 50 8
```

```bash
./runnodes.sh init 1 8
```

--------------------------------

### Configure Clone Similarity Threshold

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Adjust the clone similarity percentage by modifying the threshold value. A value of '8' corresponds to 80% similarity.

```bash
threshold="${3:-8}"
```

--------------------------------

### HTML/CSS for Page Layout

Source: https://github.com/mondego/sourcerercc/blob/master/WebApp/templates/uploader.html

Basic CSS for styling the HTML and body elements, ensuring full width and height with no overflow. This sets up the fundamental layout of the page.

```css
html,body { padding: 0pt; margin: 0; font-family:cursive; width: 100%; height: 100%; overflow-x: hidden; }
```

--------------------------------

### Navigation Links

Source: https://github.com/mondego/sourcerercc/blob/master/WebApp/templates/uploader.html

Provides navigation links for downloading a CSV file and returning to the home page. The home link uses a Jinja2 template variable for dynamic URL generation.

```html
*   [Download CSV](/getCSV)
*   [Home]({{ url_for('upload_file') }})
```

--------------------------------

### Perform Search Operation

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Modify `runnodes.sh` to initiate the search process across multiple nodes. Ensure the `threshold` variable is set and the `SEARCH_SHARD_ID` is configured in the properties file.

```bash
for i in $(seq 1 1 $num_nodes)
do
    java -Dproperties.location="$rootPATH/NODE_$i/sourcerer-cc.properties" -Xms2g -Xmx2g  -jar dist/indexbased.SearchManager.jar search $threshold &
done
```

--------------------------------

### Execute full SourcererCC pipeline with ScriptController

Source: https://context7.com/mondego/sourcerercc/llms.txt

Orchestrate the complete SourcererCC pipeline using the Python ScriptController. The pipeline can be resumed from the last successful step if it fails.

```python
# controller.py — run the full pipeline with 10 search nodes
import sys
sys.argv = ['controller.py', '10']  # num_nodes_search

from controller import ScriptController

params = {"num_nodes_search": 10}
ctrl = ScriptController(params)

try:
    ctrl.execute()
    # On success: prints "SUCCESS: Search Completed on all nodes"
    # State file: scriptinator_metadata.scc holds last completed state (0-5)
    # Logs: Log_execute_1.out, Log_init.out, Log_index.out,
    #       Log_move_index.out, Log_search.out  (+ matching .err files)
except Exception as e:
    print(f"Pipeline failed: {e}")
    # Re-run `python controller.py 10` to resume from last checkpoint

# CLI usage:
#   python controller.py 10   # run/resume full pipeline with 10 search nodes

```

--------------------------------

### DataTable Initialization with JavaScript

Source: https://github.com/mondego/sourcerercc/blob/master/WebApp/templates/uploader.html

Initializes a DataTable on an element with the ID 'data'. It configures column searchability and orderability, disabling them for specific columns.

```javascript
$(document).ready(function () { $('#data').DataTable({ columns: [ null, {searchable: false}, {orderable: false, searchable: false}, {orderable: false, searchable: false}, null,], }); });
```

--------------------------------

### Pairwise Clone Detection with CloneDetector

Source: https://context7.com/mondego/sourcerercc/llms.txt

Performs brute-force O(n²) clone detection for smaller datasets. Supports both overlap and Jaccard similarity metrics. Configure memory allocation appropriately.

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetector.jar myproject 8 overlap
```

```bash
java -Xms4g -Xmx4g -jar dist/noindex.CloneDetector.jar myproject 8 jaccard
```

--------------------------------

### Python Environment Configuration

Source: https://github.com/mondego/sourcerercc/blob/master/requirements.txt

Configure the RUN_ENVIRONMENT variable to execute the specific Python version required by the application. This is particularly important when using pyenv.

```ini
[ENVIRONMENT]
RUN_ENVIRONMENT = pyenv exec python3
```

--------------------------------

### Aggregate Clone Detection Results

Source: https://github.com/mondego/sourcerercc/blob/master/README.md

Combine the output files from multiple detection nodes into a single results file containing pairs of clone IDs.

```bash
cat clone-detector/NODE_*/output8.0/query_* > results.pairs
```

--------------------------------

### Compute clone similarity using CloneHelper.detectClones

Source: https://context7.com/mondego/sourcerercc/llms.txt

Calculate overlap or Jaccard similarity between token bags using CloneHelper. This method is central to pairwise clone detection and can operate on individual bags or sets of bags.

```java
import com.mondego.noindex.CloneHelper;
import com.mondego.models.Bag;
import java.util.HashSet;
import java.util.Set;

CloneHelper helper = new CloneHelper();
helper.setThreshold(0.8f);   // 80% similarity
helper.setTh(80f);           // internal integer representation (threshold * 100)
helper.setClonesWriter(/* java.io.Writer */ writer);

// Overlap similarity (default): count = |intersection token freqs|
//   computedThreshold = ceil(th * max(|bagA|, |bagB|) / 1000)
helper.detectClones(bagA, bagB, false);  // false = overlap

// Jaccard similarity:
//   computedThreshold = ceil(th * (|A| + |B|) / (1000 + th))
helper.detectClones(bagA, bagB, true);   // true = jaccard

// Bulk detection across all pairs (skips self-comparisons, avoids duplicate pairs)
Set<Bag> setA = new HashSet<>();
Set<Bag> setB = new HashSet<>();
helper.parseInputFileAndPopulateSet(new File("input/dataset/proj-clone-INPUT.txt"), setA);
helper.parseInputFileAndPopulateSet(new File("input/dataset/proj-clone-INPUT.txt"), setB);
helper.detectClones(setA, setB, false);

System.out.println("Total comparisons: " + helper.getComparisions());
System.out.println("Clone pairs found: " + helper.getNumClonesFound());
// Clone output written to writer in format:
//   Clones of Bag <id_A>
//   <id_B1> ,<id_B2> ...

```

--------------------------------

### Unevenly Split Query File with unevensplit.py

Source: https://context7.com/mondego/sourcerercc/llms.txt

Distributes a large query file into multiple smaller files using an arithmetic progression. This script is crucial for preparing input for distributed processing, ensuring earlier nodes receive smaller workloads.

```python
import sys, math

class Spliter:
    def __init__(self, params):
        self.split_count   = params['split_count']
        self.input_filename = params['input_filename']
        self.total_lines   = self._count_lines()
        self._compute_base_x()

    def _count_lines(self):
        with open(self.input_filename) as f:
            return sum(1 for _ in f)

    def _compute_base_x(self):
        N = self.split_count
        self.base_x = math.ceil(2 * self.total_lines / ((N+1)*(N+2)/2 - 1))

    def split(self): ...  # writes query_1.file, query_2.file, ...
```

```bash
python unevensplit.py input/dataset/blocks.file 5
```

--------------------------------

### Tokenize Java Source Code (Java)

Source: https://context7.com/mondego/sourcerercc/llms.txt

Use the Tokenizer.processMethodBody static method to strip comments, operators, and punctuation from Java source text, returning a list of identifier and keyword tokens. This is used internally by FileParser but can be called standalone.

```java
import com.mondego.parser.Tokenizer;
import org.apache.commons.io.FileUtils;
import java.io.File;
import java.util.List;

// Read a Java file and tokenize it
String source = FileUtils.readFileToString(new File("MyClass.java"), "utf-8");
List<String> tokens = Tokenizer.processMethodBody(source);

// tokens contains cleaned identifiers, keywords, type names — no operators or comments
// Example output for: "public int add(int a, int b) { return a + b; }"
// → ["public", "int", "add", "int", "a", "int", "b", "return", "a", "b"]
System.out.println("Token count: " + tokens.size());
for (String t : tokens) {
    System.out.println(t);
}
```

--------------------------------

### Merge Indexed Data

Source: https://github.com/mondego/sourcerercc/blob/master/clone-detector/README.md

Execute the `ant cdmerge` command to merge the indexed data. This process requires specific Java heap space settings.

```bash
ant cdmerge
```

--------------------------------

### Jinja2 Template Structure

Source: https://github.com/mondego/sourcerercc/blob/master/WebApp/templates/uploader.html

Defines the main content block and script block for a Jinja2 template. It includes a loop for iterating over 'test' data and displaying elements.

```html
{% extends "base.html" %} {% block content %} {% for t in test %} {% endfor %} Project1 File1 Project2 File2 {{ t[0] }} {{ t[1] }} {{ t[2] }} {{ t[3] }} {% endblock %} {% block scripts %}
```

--------------------------------

### Merge Distributed Indexes with IndexMerger

Source: https://context7.com/mondego/sourcerercc/llms.txt

Combines per-node Lucene index shards into a single unified index after all nodes have completed indexing. Can be run via a helper script or directly.

```bash
./mergeindexes.sh
```

```bash
ant cdmerge
```

```bash
java -Dproperties.location="$(pwd)/sourcerer-cc.properties" \
     -Xms6g -Xmx6g -XX:+UseCompressedOops \
     -jar dist/indexbased.IndexMerger.jar
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.