LLM as Clinical Calculator

https://github.com/stanfordaimlab/llm-as-clinical-calculator
Admin
This project explores augmenting large language models with clinician-informed tools like...
Tokens:203,412
Snippets:391
Trust Score:4.6
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Start New Experiment Command

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/README.md

Command to start a new experiment, specifying a description and the number of cases to evaluate. This command also builds the experiment and generates case data.

```bash
uv run llmcalc experiment new  --description "Evaluating 10 vignettes" --number-of-cases 1
```

--------------------------------

### Setup IPython Environment

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/get_runs.ipynb

Configures the IPython environment for automatic reloading of modules, ensuring that changes in the code are reflected immediately during interactive sessions.

```python
from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2
```

--------------------------------

### Setup Autoreload and Reload

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/send_to_annotation.ipynb

Initializes the IPython environment to automatically reload modules and sets the autoreload mode to 2, which reloads all modules before executing code. This is useful for iterative development.

```python
from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2
```

--------------------------------

### Fetch Annotated Runs with Trace Filter

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/setup_annotation.ipynb

Iterates 20 times, fetching batches of 500 runs from the DataFrame 'df' using their IDs. For each batch, it calls 'ls_client.list_runs' with a specific 'trace_filter' to get runs that have 'error classification' feedback. The results are concatenated into 'annotated_df'.

```python
annotated_df = pd.DataFrame()
start = 0
end = 500
for i in range(20):
    print(start, end, end="...")
    foo = ls_client.list_runs(
        run_ids=df[start:end].id.to_list(),
        trace_filter='eq(feedback_key, "error classification")',
    )
    fdf = pd.DataFrame()
    try:
        fdf = make_df(foo)
    except:
        pass
    annotated_df = pd.concat([annotated_df, fdf])
    start = start + 500
    end = end + 500

annotated_df.shape
```

--------------------------------

### VWO Initialization and Settings

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/ckdepi.html

Initializes Visual Website Optimizer (VWO) with account ID, version, and settings. It handles loading VWO library, managing settings from local storage, and applying element hiding styles for A/B testing. The code ensures proper execution and error handling.

```JavaScript
window._vwo_code || (function() {
    var account_id = 844388,
        version = 2.0,
        settings_tolerance = 2000,
        hide_element = 'body',
        hide_element_style = 'opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important';

    /* DO NOT EDIT BELOW THIS LINE */
    f = false,
        w = window,
        d = document,
        v = d.querySelector('#vwoCode'),
        cK = 'vwo' + account_id + '_settings',
        cc = {};
    try {
        var c = JSON.parse(localStorage.getItem('vwo' + account_id + '_config'));
        cc = c && typeof c === 'object' ? c : {};
    } catch (e) {}
    var stT = cc.stT === 'session' ? w.sessionStorage : w.localStorage;
    code = {
        use_existing_jquery: function() {
            return typeof use_existing_jquery !== 'undefined' ? use_existing_jquery : undefined;
        },
        library_tolerance: function() {
            return typeof library_tolerance !== 'undefined' ? library_tolerance : undefined;
        },
        settings_tolerance: function() {
            return cc.sT || settings_tolerance;
        },
        hide_element_style: function() {
            return '{' + (cc.hES || hide_element_style) + '}';
        },
        hide_element: function() {
            return typeof cc.hE === 'string' ? cc.hE : hide_element;
        },
        getVersion: function() {
            return version;
        },
        finish: function() {
            if (!f) {
                f = true;
                var e = d.getElementById('_vis_opt_path_hides');
                if (e) e.parentNode.removeChild(e);
            }
        },
        finished: function() {
            return f;
        },
        load: function(e) {
            var t = this.getSettings(),
                n = d.createElement('script'),
                i = this;
            if (t) {
                n.textContent = t;
                d.getElementsByTagName('head')[0].appendChild(n);
                if (!w.VWO || VWO.caE) {
                    stT.removeItem(cK);
                    i.load(e);
                }
            } else {
                n.fetchPriority = 'high';
                n.src = e;
                n.type = 'text/javascript';
                n.onerror = function() {
                    w._vwo_code.finish();
                };
                d.getElementsByTagName('head')[0].appendChild(n);
            }
        },
        getSettings: function() {
            try {
                var e = stT.getItem(cK);
                if (!e) {
                    return;
                }
                e = JSON.parse(e);
                if (Date.now() > e.e) {
                    stT.removeItem(cK);
                    return;
                }
                return e.s;
            } catch (e) {
                return;
            }
        },
        init: function() {
            if (d.URL.indexOf('vwo_disable') > -1) return;
            var e = this.settings_tolerance();
            w._vwo_settings_timer = setTimeout(function() {
                w._vwo_code.finish();
                stT.removeItem(cK);
            }, e);
            var t = d.currentScript,
                n = d.createElement('style'),
                i = this.hide_element(),
                r = t && !t.async && i ? i + this.hide_element_style() : '',
                c = d.getElementsByTagName('head')[0];
            n.setAttribute('id', '_vis_opt_path_hides');
            v && n.setAttribute('nonce', v.nonce);
            n.setAttribute('type', 'text/css');
            if (n.styleSheet) n.styleSheet.cssText = r;
            else n.appendChild(d.createTextNode(r));
            c.appendChild(n);
            this.load('https://dev.visualwebsiteoptimizer.com/j.php?a=' + account_id + '&u=' + encodeURIComponent(d.URL) + '&vn=' + version);
        }
    };
    w._vwo_code = code;
    code.init();
})();
(function() {
    var i = window;

    function t() {
        if (i._vwo_code) {
            var e = t.hidingStyle = document.getElementById('_vis_opt_path_hides') || t.hidingStyle;
            if (!i._vwo_code.finished() && !_vwo_code.libExecuted && (!i.VWO || !VWO.dNR)) {
                if (!document.getElementById('_vis_opt_path_hides')) {
                    document.getElementsByTagName('head')[0].appendChild(e);
                }
                requestAnimationFrame(t);
            }
        }
    }
    t();
})();
```

--------------------------------

### Get Specific Run ID

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/setup_annotation.ipynb

Selects a single row (index 499) from the DataFrame 'df' and extracts its 'id' into a list.

```python
df[499:500].id.to_list()
```

--------------------------------

### Generate HTML with Clickable Links to Annotated Runs

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/setup_annotation.ipynb

Creates a list of URLs from the 'app_path' in the 'revised_annotated_df', converts it to a DataFrame, and then formats the URLs into clickable HTML links. The HTML content is saved to a file named 'annotated_run_urls.html'.

```python
# used to check status of annotated runs
urls = list()
for i, row in revised_annotated_df.iterrows():
    new_row = { 'url': 'https://smith.langchain.com' + row['app_path'] }
    urls = urls + [new_row]

url_df = pd.DataFrame.from_records(urls)
filename = path_join(config.RESULTS_DATA_PATH, "annotation", f"annotated_runs.pkl")
revised_annotated_df.to_pickle(filename)
# Convert URLs to clickable HTML links
url_df['url'] = url_df['url'].apply(lambda x: f'<a href="{x}" target="_blank">{x}</a>')

# Set the 'url' column as the index to avoid duplicating it in the output
url_df_html = url_df.set_index('url')

# Generate HTML with clickable links
html_content = url_df_html.to_html(escape=False, render_links=True)

# Write the HTML content to a file
filename = path_join(config.RESULTS_DATA_PATH, "annotation", f"annotated_run_urls.html")
with open(filename, 'w') as f:
    f.write(html_content)

print(f"HTML file with clickable links has been saved to: {filename}")
```

--------------------------------

### Initialize VWO Integration

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/steroidconversion.html

This JavaScript code initializes the Visual Website Optimizer (VWO) integration for the Steroid Conversion Calculator. It sets up account details, versioning, and settings tolerance, and includes logic for hiding elements during experiment loading and managing settings via local storage.

```JavaScript
window._vwo_code || (function() { var account_id=844388, version=2.0, settings_tolerance=2000, hide_element='body', hide_element_style = 'opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important';

/* DO NOT EDIT BELOW THIS LINE */

f=false,w=window,d=document,v=d.querySelector('#vwoCode'),cK='vwo'+account_id+'_settings',cc={};
try{
    var c=JSON.parse(localStorage.getItem('vwo'+account_id+'_config'));
    cc=c&&typeof c==='object'?c:{}
}catch(e){}

var stT=cc.stT==='session'?w.sessionStorage:w.localStorage;

code={
    use_existing_jquery: function() {
        return typeof use_existing_jquery !== 'undefined' ? use_existing_jquery : undefined;
    },
    library_tolerance: function() {
        return typeof library_tolerance !== 'undefined' ? library_tolerance : undefined;
    },
    settings_tolerance: function() {
        return cc.sT || settings_tolerance;
    },
    hide_element_style: function() {
        return '{' + (cc.hES || hide_element_style) + '}';
    },
    hide_element: function() {
        return typeof cc.hE === 'string' ? cc.hE : hide_element;
    },
    getVersion: function() {
        return version;
    },
    finish: function() {
        if (!f) {
            f = true;
            var e = d.getElementById('_vis_opt_path_hides');
            if (e) e.parentNode.removeChild(e);
        }
    },
    finished: function() {
        return f;
    },
    load: function(e) {
        var t = this.getSettings(),
            n = d.createElement('script'),
            i = this;
        if (t) {
            n.textContent = t;
            d.getElementsByTagName('head')[0].appendChild(n);
            if (!w.VWO || VWO.caE) {
                stT.removeItem(cK);
                i.load(e);
            }
        } else {
            n.fetchPriority = 'high';
            n.src = e;
            n.type = 'text/javascript';
            n.onerror = function() {
                w._vwo_code.finish();
            };
            d.getElementsByTagName('head')[0].appendChild(n);
        }
    },
    getSettings: function() {
        try {
            var e = stT.getItem(cK);
            if (!e) return;
            e = JSON.parse(e);
            if (Date.now() > e.e) {
                stT.removeItem(cK);
                return;
            }
            return e.s;
        } catch (e) {
            return;
        }
    },
    init: function() {
        if (d.URL.indexOf('vwo_disable') > -1) return;
        var e = this.settings_tolerance();
        w._vwo_settings_timer = setTimeout(function() {
            w._vwo_code.finish();
            stT.removeItem(cK);
        }, e);
        var t = d.currentScript,
            n = d.createElement('style'),
            i = this.hide_element(),
            r = t && !t.async && i ? i + this.hide_element_style() : '';
        c = d.getElementsByTagName('head')[0];
        n.setAttribute('id', '_vis_opt_path_hides');
        v && n.setAttribute('nonce', v.nonce);
        n.setAttribute('type', 'text/css');
        if (n.styleSheet) n.styleSheet.cssText = r;
        else n.appendChild(d.createTextNode(r));
        c.appendChild(n);
        this.load('https://dev.visualwebsiteoptimizer.com/j.php?a=' + account_id + '&u=' + encodeURIComponent(d.URL) + '&vn=' + version);
    }
};

window._vwo_code = code;
code.init();
})();

(function() {
    var i = window;

    function t() {
        if (i._vwo_code) {
            var e = t.hidingStyle = document.getElementById('_vis_opt_path_hides') || t.hidingStyle;
            if (!i._vwo_code.finished() && !_vwo_code.libExecuted && (!i.VWO || !VWO.dNR)) {
                if (!document.getElementById('_vis_opt_path_hides')) {
                    document.getElementsByTagName('head')[0].appendChild(e);
                }
                requestAnimationFrame(t);
            }
        }
    }
    t();
})();
```

--------------------------------

### CIWA-Ar Scoring Table

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/ciwa.html

This table outlines the scoring criteria for the CIWA-Ar scale, correlating score ranges with different levels of alcohol withdrawal severity. It is used to guide clinical decision-making.

```html
<table class=\"table table-bordered table-hover\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
```

--------------------------------

### VWO Initialization and Experiment Loading

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/caprini.html

This snippet initializes the Visual Website Optimizer (VWO) on the page. It sets up account details, version information, and tolerance settings for experiments. The code dynamically loads the VWO tracking script and applies CSS to hide elements as needed for A/B testing.

```JavaScript
window._vwo_code || (function() {
    var account_id = 844388,
        version = 2.0,
        settings_tolerance = 2000,
        hide_element = 'body',
        hide_element_style = 'opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important',
        f = false,
        w = window,
        d = document,
        v = d.querySelector('#vwoCode'),
        cK = 'vwo' + account_id + '_settings',
        cc = {};
    try {
        var c = JSON.parse(localStorage.getItem('vwo' + account_id + '_config'));
        cc = c && typeof c === 'object' ? c : {};
    } catch (e) {}
    var stT = cc.stT === 'session' ? w.sessionStorage : w.localStorage;
    code = {
        use_existing_jquery: function() {
            return typeof use_existing_jquery !== 'undefined' ? use_existing_jquery : undefined;
        },
        library_tolerance: function() {
            return typeof library_tolerance !== 'undefined' ? library_tolerance : undefined;
        },
        settings_tolerance: function() {
            return cc.sT || settings_tolerance;
        },
        hide_element_style: function() {
            return '{' + (cc.hES || hide_element_style) + '}';
        },
        hide_element: function() {
            return typeof cc.hE === 'string' ? cc.hE : hide_element;
        },
        getVersion: function() {
            return version;
        },
        finish: function() {
            if (!f) {
                f = true;
                var e = d.getElementById('_vis_opt_path_hides');
                if (e) e.parentNode.removeChild(e);
            }
        },
        finished: function() {
            return f;
        },
        load: function(e) {
            var t = this.getSettings(),
                n = d.createElement('script'),
                i = this;
            if (t) {
                n.textContent = t;
                d.getElementsByTagName('head')[0].appendChild(n);
                if (!w.VWO || VWO.caE) {
                    stT.removeItem(cK);
                    i.load(e);
                }
            } else {
                n.fetchPriority = 'high';
                n.src = e;
                n.type = 'text/javascript';
                n.onerror = function() {
                    w._vwo_code.finish();
                };
                d.getElementsByTagName('head')[0].appendChild(n);
            }
        },
        getSettings: function() {
            try {
                var e = stT.getItem(cK);
                if (!e) {
                    return;
                }
                e = JSON.parse(e);
                if (Date.now() > e.e) {
                    stT.removeItem(cK);
                    return;
                }
                return e.s;
            } catch (e) {
                return;
            }
        },
        init: function() {
            if (d.URL.indexOf('vwo_disable') > -1) return;
            var e = this.settings_tolerance();
            w._vwo_settings_timer = setTimeout(function() {
                w._vwo_code.finish();
                stT.removeItem(cK);
            }, e);
            var t = d.currentScript,
                n = d.createElement('style'),
                i = this.hide_element(),
                r = t && !t.async && i ? i + this.hide_element_style() : '',
                c = d.getElementsByTagName('head')[0];
            n.setAttribute('id', '_vis_opt_path_hides');
            v && n.setAttribute('nonce', v.nonce);
            n.setAttribute('type', 'text/css');
            if (n.styleSheet) n.styleSheet.cssText = r;
            else n.appendChild(d.createTextNode(r));
            c.appendChild(n);
            this.load('https://dev.visualwebsiteoptimizer.com/j.php?a=' + account_id + '&u=' + encodeURIComponent(d.URL) + '&vn=' + version);
        }
    };
    w._vwo_code = code;
    code.init();
})();
(function() {
    var i = window;

    function t() {
        if (i._vwo_code) {
            var e = t.hidingStyle = document.getElementById('_vis_opt_path_hides') || t.hidingStyle;
            if (!i._vwo_code.finished() && !_vwo_code.libExecuted && (!i.VWO || !VWO.dNR)) {
                if (!document.getElementById('_vis_opt_path_hides')) {
                    document.getElementsByTagName('head')[0].appendChild(e);
                }
                requestAnimationFrame(t);
            }
        }
    }
    t();
})();
```

--------------------------------

### CIWA-Ar Calculator for Alcohol Withdrawal

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/ciwa.html

This section details the input fields for the CIWA-Ar calculator, which assesses various symptoms of alcohol withdrawal. Each symptom has a corresponding score range and specific questions or observations to guide the assessment.

```json
{
  "input_schema": [
    {
      "label_en": "Nausea/vomiting",
      "name": "nausea",
      "options": [
        {"label": "No nausea and no vomiting", "value": 0},
        {"label": "Mild nausea and no vomiting", "value": 1},
        {"label": "(More severe symptoms) ", "value": 2},
        {"label": "(More severe symptoms)", "value": 3},
        {"label": "Intermittent nausea with dry heaves", "value": 4},
        {"label": "(More severe symptoms)", "value": 5},
        {"label": "(More severe symptoms)", "value": 6},
        {"label": "Constant nausea, frequent dry heaves and vomiting", "value": 7}
      ],
      "tips_en": "Ask 'Do you feel sick to your stomach? Have you vomited?'",
      "type": "radio"
    },
    {
      "label_en": "Tremor",
      "name": "tremor",
      "options": [
        {"label": "No tremor", "value": 0},
        {"label": "Not visible, but can be felt fingertip to fingertip", "value": 1},
        {"label": "(More severe symptoms)", "value": 2},
        {"label": "(More severe symptoms)", "value": 3},
        {"label": "Moderate, with patient's arms extended", "value": 4},
        {"label": "(More severe symptoms)", "value": 5},
        {"label": "(Mor"
      ],
      "type": "radio"
    }
  ]
}
```

--------------------------------

### Python Project Setup for LLM Clinical Calculator

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/ckdepi.html

This snippet outlines the basic project structure and dependencies for the 'llm-as-clinical-calculator' project hosted on GitHub. It highlights the use of Python for developing LLM-based medical tools.

```Python
import os

# Project: /stanfordaimlab/llm-as-clinical-calculator

# Placeholder for project-specific configurations or imports
# Example: 
# from utils import load_config
# config = load_config('config.yaml')

print("LLM as Clinical Calculator Project Initialized")

```

--------------------------------

### Display and Select Datasets

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/get_runs.ipynb

Retrieves the names and descriptions of the last ten datasets, formats them into a readable string, and prints them. It then selects the name of the very first dataset for subsequent operations.

```python
# get last few datasets and assign a dataset_name to work with
last_few_dataset_names = [datasets.iloc[i]["name"]+ ": "+datasets.iloc[i]["description"] for i in range(10)]
last_few_dataset_names = "\n--- ".join(last_few_dataset_names)
print(f"Last few datasets:\n--- {last_few_dataset_names}")
last_dataset = datasets.iloc[0]["name"]
print(f"Last dataset: {last_dataset}")
```

--------------------------------

### Prepare Experiment Command

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/README.md

Command to prepare the system for running an experiment, typically involving rebuilding experiment configurations.

```bash
uv run llmcalc experiment rebuild
```

--------------------------------

### Function to Get Runs from Annotation Queue

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/setup_annotation.ipynb

Defines a function 'get_runs_from_q' that takes a queue name and a number of runs as input. It retrieves a specified number of runs from the given annotation queue, fetches their details, and returns them as a pandas DataFrame.

```python
def get_runs_from_q(qname, num):
    runs = pd.DataFrame()
    qid = dataset_queues_df[dataset_queues_df.name == qname].iloc[0].id
    for indx in range(0,num):
        rn = ls_client.get_run_from_annotation_queue(queue_id=qid, index=indx)
        run_id = rn.id
        single_run_df = make_df(ls_client.list_runs(run_ids=[run_id]))
        runs = pd.concat([runs, single_run_df])
    return runs
```

--------------------------------

### Initialize IPython and Auto-reload

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/analysis/setup_annotation.ipynb

Initializes the IPython environment by loading the autoreload extension and setting it to reload all modules automatically before executing code. This is useful for development to ensure changes in imported modules are reflected immediately.

```python
import numpy as np
from IPython.lib.deepreload import reload
%load_ext autoreload
%autoreload 2
```

--------------------------------

### Initialize and Load Serum Osmolality Calculator

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/external_data/raw_html/osms.html

This JavaScript code initializes a serum osmolality/osmolarity calculator. It sets up configuration, handles potential jQuery dependencies, and loads external scripts necessary for its functionality. The script also includes logic for hiding elements during initialization and managing settings via local storage.

```JavaScript
window._vwo_code || (function() {
    var account_id = 844388,
        version = 2.0,
        settings_tolerance = 2000,
        hide_element = 'body',
        hide_element_style = 'opacity:0 !important;filter:alpha(opacity=0) !important;background:none !important';

    // DO NOT EDIT BELOW THIS LINE
    f = false,
    w = window,
    d = document,
    v = d.querySelector('#vwoCode'),
    cK = 'vwo' + account_id + '_settings',
    cc = {};

    try {
        var c = JSON.parse(localStorage.getItem('vwo' + account_id + '_config'));
        cc = c && typeof c === 'object' ? c : {};
    } catch (e) {}

    var stT = cc.stT === 'session' ? w.sessionStorage : w.localStorage;

    code = {
        use_existing_jquery: function() {
            return typeof use_existing_jquery !== 'undefined' ? use_existing_jquery : undefined;
        },
        library_tolerance: function() {
            return typeof library_tolerance !== 'undefined' ? library_tolerance : undefined;
        },
        settings_tolerance: function() {
            return cc.sT || settings_tolerance;
        },
        hide_element_style: function() {
            return '{' + (cc.hES || hide_element_style) + '}';
        },
        hide_element: function() {
            return typeof cc.hE === 'string' ? cc.hE : hide_element;
        },
        getVersion: function() {
            return version;
        },
        finish: function() {
            if (!f) {
                f = true;
                var e = d.getElementById('_vis_opt_path_hides');
                if (e) e.parentNode.removeChild(e);
            }
        },
        finished: function() {
            return f;
        },
        load: function(e) {
            var t = this.getSettings(),
                n = d.createElement('script'),
                i = this;
            if (t) {
                n.textContent = t;
                d.getElementsByTagName('head')[0].appendChild(n);
                if (!w.VWO || VWO.caE) {
                    stT.removeItem(cK);
                    i.load(e);
                }
            } else {
                n.fetchPriority = 'high';
                n.src = e;
                n.type = 'text/javascript';
                n.onerror = function() {
                    w._vwo_code.finish();
                };
                d.getElementsByTagName('head')[0].appendChild(n);
            }
        },
        getSettings: function() {
            try {
                var e = stT.getItem(cK);
                if (!e) return;
                e = JSON.parse(e);
                if (Date.now() > e.e) {
                    stT.removeItem(cK);
                    return;
                }
                return e.s;
            } catch (e) {
                return;
            }
        },
        init: function() {
            if (d.URL.indexOf('vwo_disable') > -1) return;
            var e = this.settings_tolerance();
            w._vwo_settings_timer = setTimeout(function() {
                w._vwo_code.finish();
                stT.removeItem(cK);
            }, e);
            var t = d.currentScript,
                n = d.createElement('style'),
                i = this.hide_element(),
                r = t && !t.async && i ? i + this.hide_element_style() : '',
                c = d.getElementsByTagName('head')[0];
            n.setAttribute('id', '_vis_opt_path_hides');
            v && n.setAttribute('nonce', v.nonce);
            n.setAttribute('type', 'text/css');
            if (n.styleSheet) n.styleSheet.cssText = r;
            else n.appendChild(d.createTextNode(r));
            c.appendChild(n);
            this.load('https://dev.visualwebsiteoptimizer.com/j.php?a=' + account_id + '&u=' + encodeURIComponent(d.URL) + '&vn=' + version);
        }
    };

    w._vwo_code = code;
    code.init();
})();

(function() {
    var i = window;

    function t() {
        if (i._vwo_code) {
            var e = t.hidingStyle = document.getElementById('_vis_opt_path_hides') || t.hidingStyle;
            if (!i._vwo_code.finished() && !_vwo_code.libExecuted && (!i.VWO || !VWO.dNR)) {
                if (!document.getElementById('_vis_opt_path_hides')) {
                    document.getElementsByTagName('head')[0].appendChild(e);
                }
                requestAnimationFrame(t);
            }
        }
    }
    t();
})();
```

--------------------------------

### Configure Miscellaneous API Keys

Source: https://github.com/stanfordaimlab/llm-as-clinical-calculator/blob/main/sample_env.txt

Sets environment variables for other essential API keys, such as Notion and E2B, which may be used for data management or specific tool integrations within the project.

```shell
# Misc API keys
NOTION_API_KEY="{YOUR_KEY_HERE}"
E2B_API_KEY="{YOUR_KEY_HERE}"
```
Tokens:
Raw