### List generated sequence examples and copy to serving location

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

After generation, list the sequence examples in GCS using `gsutil ls -l` and then copy the third set of examples to a designated serving location using `gsutil cp`.

```bash
%bash
gsutil ls -l ${SEQEX_IN_GCS}
gsutil cp ${SERV_DS} ${SERV_LOC}
```

--------------------------------

### List Sequence Examples in GCS

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Use the gsutil command-line tool to list the size and location of sequence examples stored in Google Cloud Storage, which will be used for inference.

```bash
gsutil ls -l ${SEQEX_IN_GCS}
```

--------------------------------

### Run Example Analytical Queries

Source: https://context7.com/google/fhir/llms.txt

This bash script executes example analytical queries. It assumes the previous steps for data preparation have been completed.

```bash
# Step 4: Run example analytical queries
./examples/bigquery/04-run-queries.sh

```

--------------------------------

### Generate .prototxt Files from FHIR JSON Examples

Source: https://github.com/google/fhir/blob/master/examples/protogen/README.md

Use this script with JsonToProtoMain to convert FHIR JSON example data into .prototxt files. It defaults to using examples in fhir/testdata/stu3/examples/.

```bash
./generate-testdata.sh [-i input-dir] [-o output-dir]
```

--------------------------------

### Start TensorBoard for Model Inspection

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Launches TensorBoard to visualize training runs and model graphs. Ensure MODEL_PATH is correctly set to your model's directory.

```python
from google.datalab.ml import TensorBoard as tb
tb.start(MODEL_PATH)
```

--------------------------------

### Regenerate Go Protos

Source: https://github.com/google/fhir/blob/master/go/proto/README.md

Run this script to generate the Go protos. Ensure protoc and protoc-gen-go are installed.

```shell
./go/generate_go_protos_default.sh
```

--------------------------------

### Remove previous sequence examples

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Before regenerating sequence examples, remove any existing ones from GCS using `gsutil rm`.

```bash
%bash
gsutil rm ${SEQEX_IN_GCS}
```

--------------------------------

### Initialize and configure Apache Beam pipeline for Sequence Examples

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

This code initializes an Apache Beam pipeline, defines functions to read FHIR bundles and labels, and sets up the pipeline to generate Sequence Examples. It specifies input sources, data transformations, and output destinations in GCS.

```python
google_cloud_options.job_name = SEQEX_JOB
p1 = beam.Pipeline(options=options)

def _get_version_config(version_config_path):
  with open(version_config_path) as f:
    return text_format.Parse(f.read(), version_config_pb2.VersionConfig())

version_config = _get_version_config("/usr/local/fhir_ml/proto/stu3_version_config.textproto")

keyed_bundles = ( 
    p1 
    | 'readBundles' >> beam.io.ReadFromTFRecord(
        TF_RECORD_BUNDLES, coder=beam.coders.ProtoCoder(resources_pb2.Bundle))
    | 'KeyBundlesByPatientId' >> beam.ParDo(
        bundle_to_seqex.KeyBundleByPatientIdFn()))

event_labels = ( 
    p1 | 'readEventLabels' >> beam.io.ReadFromTFRecord(
        TF_RECORD_LABELS,
        coder=beam.coders.ProtoCoder(fhirproto_extensions_pb2.EventLabel)))

keyed_event_labels = bundle_to_seqex.CreateTriggerLabelsPairLists(
    event_labels)

bundles_and_labels = bundle_to_seqex.CreateBundleAndLabels(
    keyed_bundles, keyed_event_labels)
_ = ( 
    bundles_and_labels
    | 'GenerateSeqex' >> beam.ParDo(
        bundle_to_seqex.BundleAndLabelsToSeqexDoFn(
            version_config=version_config,
            enable_attribution=False,
            generate_sequence_label=False))
    | 'WriteSeqex' >> beam.io.WriteToTFRecord(
        SEQEX_PATH,
        coder=beam.coders.ProtoCoder(example_pb2.SequenceExample),
        file_name_suffix='.tfrecords',
        num_shards=3))
```

--------------------------------

### Initialize Apache Beam Pipeline for Label Generation

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Initializes an Apache Beam pipeline with specified Google Cloud options for processing data and generating labels. This setup is crucial before defining any data processing steps.

```python
google_cloud_options = options.view_as(GoogleCloudOptions)
google_cloud_options.project = GCP_PROJECT
google_cloud_options.job_name = LABELS_JOB
google_cloud_options.staging_location = STAGING_LOCATION
google_cloud_options.temp_location = TEMP_LOCATION
options.view_as(StandardOptions).runner = RUNNER
```

--------------------------------

### Setup Environment Variables for GCP and GCS

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Initialize environment variables for your GCP project, GCS bucket, and region. Ensure these values are set before running subsequent cells.

```python
import os
GCP_PROJECT = 'dp-workspace'
GCS_BUCKET = 'gs://cluster19-bkt'
GCS_REGION = 'us-central1'
os.putenv("REGION", GCS_REGION)
LABELS_JOB = 'bundlesTolabels'
SEQEX_JOB = 'gen_seqex'
STAGING_LOCATION = GCS_BUCKET+'/staging'
TEMP_LOCATION = GCS_BUCKET+'/temp'
RUNNER = 'DirectRunner'
TF_RECORD_BUNDLES = 'gs://cluster19-bkt/synthea/bundles/bundles*'
os.putenv("BUNDLES_IN_GCS", TF_RECORD_BUNDLES)
LABELS_PATH = GCS_BUCKET+'/synthea/train/label'
TF_RECORD_LABELS = GCS_BUCKET+'/synthea/train/label-00000-of-00001.tfrecords'
os.putenv("LABELS_IN_GCS", TF_RECORD_LABELS)
SEQEX_PATH = GCS_BUCKET+'/synthea/train/seqex'
TF_RECORD_SEQEX = GCS_BUCKET+'/synthea/train/seqex*'
os.putenv("SEQEX_IN_GCS", TF_RECORD_SEQEX)
MODEL_PATH = GCS_BUCKET+'/synthea/model/'
os.putenv("MODEL_IN_GCS", MODEL_PATH+"*")
SAVED_MODEL_PATH = MODEL_PATH + 'export'
os.putenv("SAVED_MODEL_IN_GCS", SAVED_MODEL_PATH+"*")
TRAINING_DATASET = GCS_BUCKET+'/synthea/train/seqex-00000-of-00003.tfrecords'
VALIDATION_DATASET = GCS_BUCKET+'/synthea/train/seqex-00001-of-00003.tfrecords'
SERVING_DATASET = GCS_BUCKET+'/synthea/train/seqex-00002-of-00003.tfrecords'
os.putenv("SERV_DS", SERVING_DATASET)
os.putenv("SERV_LOC", GCS_BUCKET+"/synthea/serv/")
```

--------------------------------

### Run FHIR to BigQuery Upload Process

Source: https://github.com/google/fhir/blob/master/examples/bigquery/README.md

Execute the shell scripts to download Synthea data, split FHIR bundles by resource type, upload to BigQuery, and run queries. Ensure the Cloud SDK is installed and initialized.

```bash
MY_DIR=/tmp/fhir
./01-get-synthea.sh $MY_DIR
./02-split-bundles-by-resource-type.sh $MY_DIR
./03-upload-to-bq.sh $MY_DIR
./04-run-queries.sh
```

--------------------------------

### Go: Get, Set, and Append Proto Fields with protopath

Source: https://context7.com/google/fhir/llms.txt

Use protopath.Get and protopath.Set for type-safe access to deeply nested proto fields via dot-delimited paths. Supports repeated fields by index and appending with '-1'.

```go
import (
    "fmt"
    "github.com/google/fhir/go/protopath"
    d4pb "github.com/google/fhir/go/proto/google/fhir/proto/r4/core/datatypes_go_proto"
    r4pb "github.com/google/fhir/go/proto/google/fhir/proto/r4/core/resources/patient_go_proto"
)

func main() {
    patient := &r4pb.Patient{
        Id: &d4pb.Id{Value: "pt-42"},
        Name: []*d4pb.HumanName{
            {Family: &d4pb.String{Value: "Smith"}},
        },
    }

    // Get the patient's family name from the first name entry
    family, err := protopath.Get[*d4pb.String](patient, protopath.NewPath("name.0.family"))
    if err != nil {
        panic(err)
    }
    fmt.Println(family.GetValue()) // Output: Smith

    // Get with a default when field is missing
    given, err := protopath.GetWithDefault[*d4pb.String](
        patient,
        protopath.NewPath("name.0.given.0"),
        &d4pb.String{Value: "Unknown"},
    )
    if err != nil {
        panic(err)
    }
    fmt.Println(given.GetValue()) // Output: Unknown

    // Set a deeply nested field value
    err = protopath.Set(patient, protopath.NewPath("id.value"), "pt-99")
    if err != nil {
        panic(err)
    }
    fmt.Println(patient.GetId().GetValue()) // Output: pt-99

    // Append to a repeated field using index "-1"
    newName := &d4pb.HumanName{Family: &d4pb.String{Value: "Jones"}}
    err = protopath.Set(patient, protopath.NewPath("name.-1"), newName)
    if err != nil {
        panic(err)
    }
    fmt.Println(len(patient.Name)) // Output: 2

    // Clear a field using protopath.Zero
    err = protopath.Set(patient, protopath.NewPath("id"), protopath.Zero)
    fmt.Println(patient.Id) // Output: <nil>
}
```

--------------------------------

### Import Necessary Libraries for Apache Beam and TensorFlow

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Imports required libraries for Apache Beam pipeline options, TensorFlow, and FHIR data processing. Ensure these libraries are installed in your environment.

```python
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import GoogleCloudOptions
from apache_beam.options.pipeline_options import StandardOptions
import apache_beam as beam
from tensorflow.core.example import example_pb2
import tensorflow as tf
import time

from proto import version_config_pb2
from proto.stu3 import fhirproto_extensions_pb2
from proto.stu3 import resources_pb2

from google.protobuf import text_format
from py.google.fhir.labels import label
from py.google.fhir.labels import bundle_to_label
from py.google.fhir.seqex import bundle_to_seqex
from py.google.fhir.models import model
```

--------------------------------

### Profile Generation — protogen and Custom Implementation Guides

Source: https://context7.com/google/fhir/llms.txt

FhirProto supports generating custom `.proto` files from FHIR profile definitions (JSON StructureDefinitions), enabling strongly-typed profiled resources. Profiles are defined in JSON/YAML, compiled via Bazel, and the generated protos can then be used with the same JSON/validation APIs.

```APIDOC
## Profile Generation — `protogen` and Custom Implementation Guides

FhirProto supports generating custom `.proto` files from FHIR profile definitions (JSON StructureDefinitions), enabling strongly-typed profiled resources. Profiles are defined in JSON/YAML, compiled via Bazel, and the generated protos can then be used with the same JSON/validation APIs.

```bash
# 1. Define your profile in examples/profiles/demo.json (StructureDefinition JSON)
# 2. Add a BUILD target:
#    proto_library(name = "demo_proto", ...)
#    fhir_package(name = "demo", srcs = ["demo.json"])

# 3. Generate .proto files from profile definitions
./generate_definitions_and_protos.sh //examples/profiles:demo

# 4. Build the generated protos
bazel build //examples/profiles:demo_go_proto

# 5. Use with the standard C++ profiler to convert base resources to profiled form
bazel run //examples/profiles:LocalProfiler -- /tmp/fhir

# 6. Generate BigQuery schema for the profiled resource
bazel run //java:BigQuerySchemaGenerator -- /tmp/fhir

# 7. Upload profiled NDJSON to BigQuery
bq load --source_format=NEWLINE_DELIMITED_JSON \
    --schema=/tmp/fhir/DemoPatient.schema.json \
    synthea.DemoPatient \
    /tmp/fhir/DemoPatient.ndjson
```
```

--------------------------------

### Initialize Apache Beam Pipeline Options

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Sets up the basic pipeline options for Apache Beam. Further configurations can be added as needed.

```python
options = PipelineOptions()

```

--------------------------------

### Prepare GCP resources for Dataproc cluster

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

Run this script to move cluster initialization scripts to a Google Cloud Storage bucket. Optionally, create a persistent Hive Metastore.

```bash
./01-prep.sh {hivemeta}
```

--------------------------------

### Generate DescriptorProtos as .prototxt Files

Source: https://github.com/google/fhir/blob/master/examples/protogen/README.md

Execute this script to create DescriptorProtos in .prototxt format. Input and output directories can be customized.

```bash
./generate-descriptors.sh [-i input-dir] [-o output-dir]
```

--------------------------------

### Import Apache Beam Pipeline Options (Commented Out)

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

This commented-out code block shows the necessary imports for setting up Apache Beam pipeline options, including Google Cloud and Standard options.

```python
# from apache_beam.options.pipeline_options import PipelineOptions
# from apache_beam.options.pipeline_options import GoogleCloudOptions
# from apache_beam.options.pipeline_options import StandardOptions
```

--------------------------------

### TensorFlow Model Function Call Logging

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Logs messages indicating the start and end of the TensorFlow model_fn execution, which is a key part of the model training and serving process.

```python
I0306 22:31:09.620572 140440319678208 tf_logging.py:115] Calling model_fn.
```

```python
I0306 22:31:09.624799 140440319678208 tf_logging.py:115] Calling model_fn.
```

```python
I0306 22:31:12.750710 140440319678208 tf_logging.py:115] Done calling model_fn.
```

```python
I0306 22:31:12.755114 140440319678208 tf_logging.py:115] Done calling model_fn.
```

--------------------------------

### Source environment file for GCP provisioning

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

Source this file to set up your environment variables for GCP provisioning scripts. Ensure the bucket name is unique.

```bash
eg. source ./myenv.sh
```

--------------------------------

### JSON Unmarshalling with Validation

Source: https://context7.com/google/fhir/llms.txt

Creates a validating `Unmarshaller` that parses FHIR JSON bytes into a version-specific `ContainedResource` proto. It performs full FHIR validation, including required fields, reference types, and primitive regexes. The example demonstrates unmarshalling a Patient resource and extracting the concrete Patient message.

```APIDOC
## Unmarshal FHIR JSON with Validation

### Description
Parses FHIR JSON bytes into a version-specific `ContainedResource` proto, performing full FHIR validation.

### Method
`jsonformat.NewUnmarshaller(timezone string, version fhirversion.Version) (*Unmarshaller, error)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```go
import (
    "fmt"
    "github.com/google/fhir/go/fhirversion"
    "github.com/google/fhir/go/jsonformat"
    r4pb "github.com/google/fhir/go/proto/google/fhir/proto/r4/core/resources/patient_go_proto"
    "google.golang.org/protobuf/proto"
)

func main() {
    // Create a validating unmarshaller for FHIR R4, using UTC as the default timezone
    um, err := jsonformat.NewUnmarshaller("UTC", fhirversion.R4)
    if err != nil {
        panic(err)
    }

    patientJSON := []byte(`{
        "resourceType": "Patient",
        "id": "example-patient",
        "name": [{"family": "Doe", "given": ["John"]}],
        "birthDate": "1990-01-15",
        "gender": "male"
    }`)

    // Unmarshal into a ContainedResource proto
    containedRes, err := um.Unmarshal(patientJSON)
    if err != nil {
        // err may be an UnmarshalErrorList with per-field diagnostics
        fmt.Printf("validation error: %v\n", err)
        return
    }

    // Extract the concrete Patient message
    cr := containedRes.ProtoReflect()
    od := cr.Descriptor().Oneofs().ByName("oneof_resource")
    fd := cr.WhichOneof(od)
    patient := cr.Get(fd).Message().Interface().(*r4pb.Patient)
    fmt.Println("Patient ID:", patient.GetId().GetValue())
    // Output: Patient ID: example-patient
}
```

### Response
#### Success Response (200)
- **ContainedResource** (proto message) - The unmarshalled FHIR resource.

#### Response Example
(See Go code for extraction logic)
```

--------------------------------

### Initialize Environment Variables for CloudML

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Set up essential environment variables for your GCP project, GCS bucket, region, and model paths. Ensure 'GCP_PROJECT' and 'GCS_BUCKET' are updated before execution.

```python
import os
GCP_PROJECT = 'dp-workspace'
GCS_BUCKET = 'gs://cluster19-bkt'
GCS_REGION = 'us-central1'
os.putenv("REGION", GCS_REGION)
TF_RECORD_SEQEX = GCS_BUCKET+'/synthea/serv/seqex*'
os.putenv("SEQEX_IN_GCS", TF_RECORD_SEQEX)
MODEL_PATH = GCS_BUCKET+'/synthea/model/'
os.putenv("MODEL_IN_GCS", MODEL_PATH+"*")
SAVED_MODEL_PATH = MODEL_PATH + 'export'
os.putenv("SAVED_MODEL_IN_GCS", SAVED_MODEL_PATH+"*")
SERVING_DATASET = GCS_BUCKET+'/synthea/serv/seqex-00002-of-00003.tfrecords'
os.putenv("SERVING_DATASET", SERVING_DATASET)
INFERENCE_PATH = MODEL_PATH + 'infer'
os.putenv("INFERENCE_PATH", INFERENCE_PATH)
os.putenv("MODEL_NAME", "tf_fhir_los")
```

--------------------------------

### Launch Jupyter Notebook on Linux for Cloud Datalab

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

This script opens a Jupyter notebook within a Chrome web browser on a Linux workstation, providing access to Cloud Datalab. It requires an environment file.

```bash
./jupyterconnect-linux.sh {your env file}
```

--------------------------------

### Generate .proto files

Source: https://github.com/google/fhir/blob/master/examples/profiles/README.md

Generates .proto files required for profile definition. This is a prerequisite for subsequent data conversion steps.

```bash
# Generate .proto files
./generate_definitions_and_protos.sh //examples/profiles:demo
```

--------------------------------

### TensorFlow Model Configuration and Logging

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

This snippet shows the configuration details for a TensorFlow model, including checkpointing, session configuration, and distributed training settings. It also logs the configuration used.

```python
I0306 22:31:04.621371 140440319678208 tf_logging.py:115] Using config: {
  "_save_checkpoints_secs": 180,
  "_session_config": allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
,
  "_keep_checkpoint_max": 5,
  "_task_type": None,
  "_train_distribute": None,
  "_is_chief": True,
  "_cluster_spec": <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fba7c8554d0>,
  "_tf_config": gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
,
  "_protocol": None,
  "_save_checkpoints_steps": None,
  "_keep_checkpoint_every_n_hours": 10000,
  "_num_ps_replicas": 0,
  "_model_dir": "gs://cluster19-bkt/synthea/model/",
  "_tf_random_seed": None,
  "_master": "",
  "_device_fn": None,
  "_num_worker_replicas": 0,
  "_task_id": 0,
  "_log_step_count_steps": 100,
  "_evaluation_master": "",
  "_eval_distribute": None,
  "_environment": "local",
  "_save_summary_steps": 100
}
```

--------------------------------

### List Deployed Models in Cloud ML Engine

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Use this command to view all models currently deployed in your Cloud ML Engine project. This helps in managing existing deployments.

```bash
gcloud ml-engine models list
```

--------------------------------

### Run BigQuery Query

Source: https://github.com/google/fhir/blob/master/examples/profiles/README.md

Executes a sample SQL query against the uploaded DemoPatient data in BigQuery. This demonstrates data retrieval and aggregation.

```sql
# Run a query
bq query --nouse_legacy_sql "\
  SELECT \
    birthPlace.city, \
    APPROX_TOP_COUNT(SUBSTR(mothersMaidenName, 0, 3), 2), \
    count(*) \
  FROM \
    synthea.DemoPatient p \
  GROUP BY 1 \
  ORDER BY 3 DESC \
"
```

--------------------------------

### Create a Dataproc cluster with Cloud Datalab

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

This script creates a 3-node Dataproc cluster in GCP, provisioning Cloud Datalab on the master node. It can optionally use a Hive Metastore.

```bash
./03-cluster.sh {hivemeta}
```

--------------------------------

### Launch Jupyter Notebook on MAC for Cloud Datalab

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

This script opens a Jupyter notebook within a Chrome web browser on a MAC operating system, facilitating access to Cloud Datalab. It requires an environment file.

```bash
./jupyterconnect-mac.sh {your env file}
```

--------------------------------

### Run the pipeline and measure execution time

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Execute the Apache Beam pipeline and measure the total time taken for the process. The output will be the duration in seconds.

```python
logger.setLevel(logging.CRITICAL)
start = time.time()
p1.run().wait_until_finish()
end = time.time()
print(end-start)
```

--------------------------------

### Generate STU3 FHIR Proto Definitions

Source: https://github.com/google/fhir/blob/master/examples/protogen/README.md

Run this script to generate Proto definition (.proto) files from STU3 StructureDefinitions. Specify input and output directories if defaults are not suitable.

```bash
./generate-proto.sh [-i input-dir] [-o output-dir]
```

--------------------------------

### List generated labels in GCS

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Use `gsutil ls -l` to list the generated labels stored in Google Cloud Storage.

```bash
%bash
gsutil ls -l ${LABELS_IN_GCS}
```

--------------------------------

### Build FHIR code repositories

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

Execute this script to build the FHIR code repositories. This step is needed on a per-need basis.

```bash
./02-gen-fhirdependencies.sh
```

--------------------------------

### Generate FHIR proto files from profiles

Source: https://context7.com/google/fhir/llms.txt

Use `protogen` and Bazel build rules to generate `.proto` files from FHIR StructureDefinition JSON. This enables strongly-typed profiled resources.

```bash
# 1. Define your profile in examples/profiles/demo.json (StructureDefinition JSON)
# 2. Add a BUILD target:
#    proto_library(name = "demo_proto", ...)
#    fhir_package(name = "demo", srcs = ["demo.json"])

# 3. Generate .proto files from profile definitions
./generate_definitions_and_protos.sh //examples/profiles:demo

# 4. Build the generated protos
bazel build //examples/profiles:demo_go_proto

# 5. Use with the standard C++ profiler to convert base resources to profiled form
bazel run //examples/profiles:LocalProfiler -- /tmp/fhir

# 6. Generate BigQuery schema for the profiled resource
bazel run //java:BigQuerySchemaGenerator -- /tmp/fhir

# 7. Upload profiled NDJSON to BigQuery
bq load --source_format=NEWLINE_DELIMITED_JSON \
    --schema=/tmp/fhir/DemoPatient.schema.json \
    synthea.DemoPatient \
    /tmp/fhir/DemoPatient.ndjson

```

--------------------------------

### Create a New Model in Cloud ML Engine

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

This command creates a new model in Cloud ML Engine if it does not already exist. Specify the model name and the desired region for deployment.

```bash
gcloud ml-engine models create $MODEL_NAME --regions=$REGION
```

--------------------------------

### Build R4 FHIR Java Library

Source: https://github.com/google/fhir/blob/master/java/com/google/fhir/release/README.md

Use this command to build the R4 FHIR Java library JAR file. This is for projects not using Bazel.

```bash
$ bazel build //java/com/google/fhir/release:r4_lib.jar
```

--------------------------------

### Go — JSON Marshalling (`jsonformat.NewMarshaller`)

Source: https://context7.com/google/fhir/llms.txt

Creates a `Marshaller` that serializes FHIR resource protos to standard FHIR JSON or analytics JSON, with optional pretty-printing.

```APIDOC
## Go — JSON Marshalling (`jsonformat.NewMarshaller`)

Creates a `Marshaller` that serializes a `ContainedResource` or individual FHIR resource proto to standard FHIR JSON or analytics (BigQuery-friendly) JSON, with optional pretty-printing.

### Usage Examples:

1.  **Pure FHIR JSON (compact):**
    ```go
    m, err := jsonformat.NewMarshaller(false, "", "", fhirversion.R4)
    // ... handle error ...
    jsonBytes, err := m.Marshal(cr)
    // ... handle error ...
    fmt.Println(string(jsonBytes))
    ```

2.  **Pretty JSON:**
    ```go
    pm, _ := jsonformat.NewPrettyMarshaller(fhirversion.R4)
    prettyJSON, _ := pm.MarshalToString(cr)
    fmt.Println(prettyJSON)
    ```

3.  **Analytics JSON (for BigQuery / SQL-on-FHIR), max depth 2:**
    ```go
    am, _ := jsonformat.NewAnalyticsMarshaller(2, fhirversion.R4)
    analyticsJSON, _ := am.MarshalResourceToString(patient)
    fmt.Println(analyticsJSON)
    ```

4.  **Analytics with inferred schema (extensions as first-class fields):**
    ```go
    aim, _ := jsonformat.NewAnalyticsMarshallerWithInferredSchema(3, fhirversion.R4)
    inferredJSON, _ := aim.MarshalResourceToString(patient)
    fmt.Println(inferredJSON)
    ```
```

--------------------------------

### List Versions of a Model

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Use this command to check all deployed versions associated with a specific model in Cloud ML Engine. This is helpful for tracking different iterations of your model.

```bash
gcloud ml-engine versions list --model ${MODEL_NAME}
```

--------------------------------

### Read and Print Training Data Batch

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Reads a batch of data using the training input function and prints it to verify data loading. This helps confirm that the data pipeline is functioning correctly.

```python
map_, label_ = train_input_fn()
success = False
with tf.train.MonitoredSession() as sess:
  map_['label'] = label_
  print(sess.run(map_))
  print("Successfully read an input batch")
```

--------------------------------

### Upload DemoPatient to BigQuery

Source: https://github.com/google/fhir/blob/master/examples/profiles/README.md

Loads the converted DemoPatient data into a BigQuery table. Requires the generated schema file and the newline-delimited JSON data.

```bash
# Upload DemoPatient
bq load --source_format=NEWLINE_DELIMITED_JSON --schema=$MY_DIR/DemoPatient.schema.json synthea.DemoPatient $MY_DIR/DemoPatient.ndjson
```

--------------------------------

### Generate TFRecord training dataset

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

This script converts the previously created training data into TFRecord file format and uploads it to a GCS Bucket. Specify a scratch directory for temporary files.

```bash
./05-gen-bundles.sh {scratch directory}
```

--------------------------------

### List TFRecord Bundles in GCS

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Uses `gsutil` to list the contents of the GCS bucket containing TFRecord bundles. This helps in verifying the data is accessible.

```bash
%bash
gsutil ls -l ${BUNDLES_IN_GCS}
```

--------------------------------

### Generate FHIR training data

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

Run this script to create training data (JSON FHIR bundles) using Synthea and upload it to a GCS Bucket in your GCP project. Specify a scratch directory for temporary files.

```bash
./04-create-training-data.sh {scratch directory}
```

--------------------------------

### Define Apache Beam Pipeline for Label Generation

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Defines an Apache Beam pipeline to read TFRecord bundles, transform them into length of stay labels using a custom DoFn, and write the generated labels to another TFRecord file. Ensure necessary protobuf definitions are available.

```python
p = beam.Pipeline(options=options)

bundles = p | 'read' >> beam.io.ReadFromTFRecord(
    TF_RECORD_BUNDLES, coder=beam.coders.ProtoCoder(resources_pb2.Bundle))

labels = bundles | 'BundleToLabel' >> beam.ParDo(
    bundle_to_label.LengthOfStayRangeLabelAt24HoursFn(for_synthea=True))

_ = labels | beam.io.WriteToTFRecord(
    LABELS_PATH,
    coder=beam.coders.ProtoCoder(fhirproto_extensions_pb2.EventLabel),
    file_name_suffix='.tfrecords')
```

--------------------------------

### Create a New Model Version

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

This command creates a new version for your model in Cloud ML Engine. Ensure you increment the version number (e.g., v1, v2) for tracking. Specify the model name, the origin of the model binary, and the runtime version.

```bash
#gcloud ml-engine versions delete v1 --model ${MODEL_NAME} -q
gcloud ml-engine versions create v1 \
    --model ${MODEL_NAME} \
    --origin ${MODEL_BINARY} \
    --runtime-version 1.12
```

--------------------------------

### Convert NDJSON for BigQuery using Java Tool

Source: https://context7.com/google/fhir/llms.txt

This command uses the bazel run command to execute a Java tool for converting NDJSON files into a format suitable for BigQuery analytics. It requires input and output paths, and the FHIR version.

```bash
# To convert NDJSON for BigQuery using the Java tool (analytic JSON format):
bazel run //java:ConvertNdJsonForBigQuery -- \
    --input_path=$MY_DIR/Patient.ndjson \
    --output_path=$MY_DIR/Patient.analytic.ndjson \
    --fhir_version=STU3

```

--------------------------------

### TensorFlow Estimator Configuration Info

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

This output provides information about the TensorFlow estimator's configuration, including checkpoint saving settings, session configuration, model directory, and distribution strategies. It's useful for understanding how the model will be managed during training.

```text
INFO:tensorflow:Using config: {'_save_checkpoints_secs': 180, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_task_type': None, '_train_distribute': None, '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f0d102c9a90>, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_protocol': None, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_num_ps_replicas': 0, '_model_dir': 'gs://cluster19-bkt/synthea/model/', '_tf_random_seed': None, '_master': '', '_device_fn': None, '_num_worker_replicas': 0, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, '_environment': 'local', '_save_summary_steps': 100}

```

--------------------------------

### Convert Patient to DemoPatient

Source: https://github.com/google/fhir/blob/master/examples/profiles/README.md

Converts existing Patient data into the DemoPatient profile format. Ensure the profile name matches if modified.

```bash
# Convert Patient to DemoPatient (modify if you change name from DemoPatient)
bazel run //examples/profiles:LocalProfiler $MY_DIR
```

--------------------------------

### Enable Logging for Debugging

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Optionally enable logging for debugging purposes. Uncomment the INFO level to see more detailed logs.

```python
import logging
logger = logging.getLogger()
#logger.setLevel(logging.INFO)
logger.setLevel(logging.ERROR)
```

--------------------------------

### Run Apache Beam Pipeline and Measure Execution Time

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Executes the defined Apache Beam pipeline and measures the total time taken for the pipeline to complete. This snippet also configures logging to INFO level.

```python
logger.setLevel(logging.INFO)
start = time.time()
p.run().wait_until_finish()
end = time.time()
print(end-start)
```

--------------------------------

### Import TensorFlow and Model Utilities

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Imports necessary TensorFlow libraries and custom model utilities for creating hyperparameters, input functions, and estimators. Resets the TensorFlow graph to ensure a clean state.

```python
import tensorflow as tf
from py.google.fhir.models import model
from py.google.fhir.models.model import create_hparams
from py.google.fhir.models.model import get_input_fn
from py.google.fhir.models.model import make_estimator

tf.reset_default_graph()
hparams = model.create_hparams()

time_crossed_features = [
        cross.split(':') for cross in hparams.time_crossed_features if cross
    ]

train_input_fn = get_input_fn(tf.estimator.ModeKeys.TRAIN, TRAINING_DATASET, 'label.length_of_stay_range.class',
                              True, hparams.time_windows, hparams.include_age, hparams.categorical_context_features,
                              hparams.sequence_features, time_crossed_features, batch_size=24)
validation_input_fn = get_input_fn(tf.estimator.ModeKeys.EVAL, VALIDATION_DATASET, 'label.length_of_stay_range.class',
                                   True, hparams.time_windows, hparams.include_age, hparams.categorical_context_features,
                                   hparams.sequence_features, time_crossed_features, batch_size=24)
```

--------------------------------

### Delete Previously Deployed Model and Version

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/2_deploy_and_run_ml_model_to_predict_los.ipynb

Execute these commands to remove an existing model and its version from Cloud ML Engine. This is useful for cleaning up old deployments before creating new ones. The `-q` flag suppresses confirmation prompts.

```bash
gcloud ml-engine versions delete v1 --model ${MODEL_NAME} -q
gcloud ml-engine models delete $MODEL_NAME -q
```

--------------------------------

### TensorFlow Logging Output

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

These snippets represent standard TensorFlow logging messages during graph finalization and local initialization.

```text
INFO:tensorflow:Graph was finalized.

```

```text
I0306 22:24:15.063673 139695413741312 tf_logging.py:115] Graph was finalized.

```

```text
INFO:tensorflow:Running local_init_op.

```

```text
I0306 22:24:15.674421 139695413741312 tf_logging.py:115] Running local_init_op.

```

```text
INFO:tensorflow:Done running local_init_op.

```

```text
I0306 22:24:15.743711 139695413741312 tf_logging.py:115] Done running local_init_op.

```

--------------------------------

### TensorFlow Estimator Configuration Warning

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

This output indicates a deprecation warning from TensorFlow regarding the use of `tf.contrib.learn.python.learn.estimators.run_config`. It suggests updating to `tf.estimator.RunConfig` for future compatibility.

```text
WARNING:tensorflow:From /usr/local/fhir/py/google/fhir/models/model.py:684: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version.
Instructions for updating:
When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead.

```

--------------------------------

### Open SSH Tunnel to Cloud Datalab Master Node

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/README.md

This script opens an SSH tunnel to the master node, enabling secure access to Cloud Datalab via a Chrome web browser. It requires an environment file.

```bash
./sshtunnel.sh {your env file}
```

--------------------------------

### Go: Validate FHIR Proto Messages using fhirvalidate.Validate

Source: https://context7.com/google/fhir/llms.txt

Use `fhirvalidate.Validate` to check FHIR proto messages against spec rules, including primitive constraints and required fields. Supports custom `ErrorReporter` for structured output and offers a fast path for primitive validation.

```go
import (
    "fmt"
    "github.com/google/fhir/go/jsonformat/errorreporter"
    "github.com/google/fhir/go/jsonformat/fhirvalidate"
    "github.com/google/fhir/go/fhirversion"
    d4pb "github.com/google/fhir/go/proto/google/fhir/proto/r4/core/datatypes_go_proto"
    r4pb "github.com/google/fhir/go/proto/google/fhir/proto/r4/core/resources/patient_go_proto"
)

func main() {
    // Valid patient — no errors
    validPatient := &r4pb.Patient{
        Id: &d4pb.Id{Value: "patient-1"},
    }
    if err := fhirvalidate.Validate(validPatient); err != nil {
        fmt.Println("unexpected error:", err)
    } else {
        fmt.Println("valid patient") // Output: valid patient
    }

    // Patient with an invalid Id (contains illegal characters per FHIR regex)
    invalidPatient := &r4pb.Patient{
        Id: &d4pb.Id{Value: "bad id with spaces!"},
    }
    if err := fhirvalidate.Validate(invalidPatient); err != nil {
        fmt.Println("validation error:", err)
    }

    // Validate with a custom ErrorReporter that collects errors into OperationOutcome
    er := errorreporter.NewOperationErrorReporter(fhirversion.R4)
    if err := fhirvalidate.ValidateWithErrorReporter(invalidPatient, er); err != nil {
        fmt.Println("fatal error:", err)
    }
    for _, issue := range er.Outcome.R4Outcome.GetIssue() {
        fmt.Printf("path=%v, msg=%v\n",
            issue.Expression[0].GetValue(),
            issue.GetDiagnostics().GetValue(),
        )
    }

    // Validate only primitives (fast path, skip required-field checks)
    if err := fhirvalidate.ValidatePrimitives(invalidPatient); err != nil {
        fmt.Println("primitive error:", err)
    }

    // Disallow null required fields during unmarshalling validation
    _ = fhirvalidate.DisallowNullRequiredField()
}
```

--------------------------------

### List GCS Model Storage Location

Source: https://github.com/google/fhir/blob/master/examples/gcp_datalab/notebooks/1_train_and_eval_ml_model_to_predict_los.ipynb

Uses the gsutil command-line tool to list the contents and details of the GCS bucket where the model is stored. Requires MODEL_IN_GCS to be set.

```bash
%bash
gsutil ls -l ${MODEL_IN_GCS}
```