### StringDomainOrBuilder Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/StringDomain Documentation for the StringDomainOrBuilder interface, used for building StringDomain configurations. ```APIDOC StringDomainOrBuilder: Description: Interface for building StringDomain configurations. ``` -------------------------------- ### Install Scio REPL via Homebrew Source: https://spotify.github.io/scio/extras/Scio-REPL Installs the Scio REPL using the spotify/public Homebrew tap. This is a convenient method for Mac users to get started quickly. ```shell brew tap spotify/public brew install scio scio-repl ``` -------------------------------- ### Get Example Weight (MultiClassClassification) Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/MultiClassClassification Retrieves the example weight column name. This field specifies which column in the dataset should be used as the weight for individual examples. It is stored as a string. ```APIDOC getExampleWeight(): String Description: The weight column. Proto Field: string example_weight = 2; Returns: The exampleWeight. Definition Classes: MultiClassClassification → MultiClassClassificationOrBuilder Annotations: @Override() ``` ```APIDOC getExampleWeightBytes(): ByteString Description: Retrieves the example weight column name as a ByteString. This is an optimized representation for string data in Protocol Buffers. Proto Field: string example_weight = 2; Returns: The bytes for exampleWeight. Definition Classes: MultiClassClassification → MultiClassClassificationOrBuilder Annotations: @Override() ``` -------------------------------- ### StringDomainOrBuilder API Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/SquaredHingeOrBuilder API documentation for StringDomainOrBuilder, enabling the creation and modification of StringDomain configurations. ```APIDOC StringDomainOrBuilder: Interface for building and accessing StringDomain objects. Related: StringDomain ``` -------------------------------- ### MIDDomain Object Conversion and Utility Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/MIDDomain Includes methods for creating builder instances, converting the object to a string representation, and standard Java/Scala object lifecycle management methods like wait and writeReplace. ```APIDOC MIDDomain Object Conversion and Utilities: toBuilder(): Builder - Returns a builder instance for creating or modifying the MIDDomain object. - Returns: A Builder object. toString(): String - Returns a string representation of the MIDDomain object. - Returns: A String. wait(): Unit - Causes the current thread to wait until it is awakened, typically by a notify() or notifyAll() method. - Throws: java.lang.InterruptedException if the current thread is interrupted while waiting. wait(timeout: Long, nanos: Int): Unit - Causes the current thread to wait until it is awakened or the specified timeout expires. - Parameters: - timeout: The maximum time to wait in milliseconds. - nanos: Additional time to wait in nanoseconds. - Throws: java.lang.InterruptedException if the current thread is interrupted while waiting. wait(timeout: Long): Unit - Causes the current thread to wait until it is awakened or the specified timeout expires. - Parameters: - timeout: The maximum time to wait in milliseconds. - Throws: java.lang.InterruptedException if the current thread is interrupted while waiting. writeReplace(): Object - Replaces this object with another object during serialization, often used for versioning or proxying. - Returns: The replacement object. - Throws: java.io.ObjectStreamException if an error occurs during replacement. ``` -------------------------------- ### Get AvgNumValues - TensorFlow Metadata Scala Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/WeightedCommonStatistics Retrieves the average number of values per example, weighted by the number of examples. This metric is calculated as the total number of values divided by the number of non-missing examples. It is a double-precision floating-point number. ```scala def getAvgNumValues(): Double ``` ```protobuf double avg_num_values = 3; ``` -------------------------------- ### Scio Sparse Join Setup Source: https://spotify.github.io/scio/examples/JoinExamples.scala Initializes the Scio context and arguments, and imports necessary utilities for the sparse join example. This includes importing JoinUtil for helper functions and magnolify.guava.auto for Bloom filter encoders. ```scala val (sc, args) = ContextAndArgs(cmdlineArgs) import JoinUtil._ import magnolify.guava.auto._ ``` -------------------------------- ### TensorFlow Metadata InfinityNorm Class Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/InfinityNorm This entry consolidates documentation for key methods of the InfinityNorm class. It details method signatures, parameter types, return values, and their roles within the generated message framework. This includes methods for reflection, initialization checks, builder creation, and parsing unknown fields. ```APIDOC InfinityNorm Class Methods: internalGetMapFieldReflection(fieldNumber: Int): MapFieldReflectionAccessor - Description: Retrieves the map field reflection accessor for a given field number. - Parameters: - fieldNumber: The integer identifier for the map field. - Returns: A MapFieldReflectionAccessor object. - Attributes: protected[protobuf] - Definition Classes: InfinityNorm → GeneratedMessageV3 isInitialized(): Boolean - Description: Checks if the message has been fully initialized. - Returns: True if initialized, False otherwise. - Definition Classes: InfinityNorm → GeneratedMessageV3 → AbstractMessage → MessageLiteOrBuilder - Annotations: @Override() isInstanceOf[T0]: Boolean - Description: Checks if the object is an instance of a specified type T0. - Returns: True if the object is an instance of T0, False otherwise. - Definition Classes: Any makeExtensionsImmutable(): Unit - Description: Makes the extensions immutable, typically called during message finalization. - Returns: Unit (no return value). - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 ne(arg0: AnyRef): Boolean - Description: Compares this object to the specified object for inequality. - Parameters: - arg0: The object to compare with. - Returns: True if the objects are not equal, False otherwise. - Definition Classes: AnyRef newBuilderForType(parent: BuilderParent): Builder - Description: Creates a new builder for the message type, associated with a parent builder. - Parameters: - parent: The parent builder context. - Returns: A new Builder instance for InfinityNorm. - Attributes: protected[v0] - Definition Classes: InfinityNorm → GeneratedMessageV3 - Annotations: @Override() newBuilderForType(): Builder - Description: Creates a new builder for the message type. - Returns: A new Builder instance for InfinityNorm. - Definition Classes: InfinityNorm → Message → MessageLite - Annotations: @Override() newBuilderForType(parent: AbstractMessage.BuilderParent): Message.Builder - Description: Creates a new builder for the message type, accepting an AbstractMessage.BuilderParent. - Parameters: - parent: The parent builder context. - Returns: A new Message.Builder instance. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 → AbstractMessage newInstance(unused: UnusedPrivateParameter): AnyRef - Description: Creates a new instance of the message, often used internally. - Parameters: - unused: An unused parameter, typically a placeholder. - Returns: A new AnyRef instance. - Attributes: protected[v0] - Definition Classes: InfinityNorm → GeneratedMessageV3 - Annotations: @Override(), @SuppressWarnings() notify(): Unit - Description: Wakes up a single thread that is waiting on this object's monitor. - Returns: Unit. - Definition Classes: AnyRef - Annotations: @native() notifyAll(): Unit - Description: Wakes up all threads that are waiting on this object's monitor. - Returns: Unit. - Definition Classes: AnyRef - Annotations: @native() parseUnknownField(input: CodedInputStream, unknownFields: Builder, extensionRegistry: ExtensionRegistryLite, tag: Int): Boolean - Description: Parses an unknown field from the input stream. - Parameters: - input: The CodedInputStream to read from. - unknownFields: The builder for unknown fields. - extensionRegistry: The ExtensionRegistryLite for handling extensions. - tag: The tag of the unknown field. - Returns: True if the field was successfully parsed, False otherwise. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 ``` -------------------------------- ### JensenShannonDivergenceOrBuilder API Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/JensenShannonDivergenceOrBuilder Documentation for the JensenShannonDivergenceOrBuilder interface, covering methods to inspect fields and the initialization state of the builder. ```APIDOC JensenShannonDivergenceOrBuilder: getUnknownFields(): UnknownFieldSet - Returns the unknown fields associated with the builder. - Returns: The UnknownFieldSet. hasField(field: FieldDescriptor): Boolean - Checks if the specified field is set. - Parameters: - field: The FieldDescriptor to check. - Returns: True if the field is set, false otherwise. hasOneof(oneof: OneofDescriptor): Boolean - Checks if any field within the specified oneof is set. - Parameters: - oneof: The OneofDescriptor to check. - Returns: True if any field in the oneof is set, false otherwise. hasSource(): Boolean - Checks if the optional source field is set. - Returns: True if the source field is set, false otherwise. hasThreshold(): Boolean - Checks if the optional threshold field is set. - Returns: True if the threshold field is set, false otherwise. isInitialized(): Boolean - Checks if the message is initialized (all required fields are set). - Returns: True if the message is initialized, false otherwise. ``` -------------------------------- ### Read Binary Files with Custom Reader Source: https://spotify.github.io/scio/io/Binary Demonstrates how to read binary files using Scio's `binaryFile` method. It requires a custom `BinaryFileReader` to parse the file content. The example shows the basic setup and points to a more comprehensive example for detailed implementation. ```scala import com.spotify.scio.ScioContext import com.spotify.scio.io.BinaryIO.BinaryFileReader val sc: ScioContext = ??? val myBinaryFileReader: BinaryFileReader = ??? sc.binaryFile("gs://", myBinaryFileReader) ``` -------------------------------- ### Schema: getDatasetConstraints Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/Schema Provides access to dataset-level constraints, such as information about changes in the number of examples. Includes methods to get the constraints directly or via an OrBuilder. ```APIDOC Schema: getDatasetConstraints(): DatasetConstraints - Description: Dataset-level constraints. This is currently used for specifying information about changes in num_examples. - Returns: The datasetConstraints. - Protobuf Definition: optional .tensorflow.metadata.v0.DatasetConstraints dataset_constraints = 11; getDatasetConstraintsOrBuilder(): DatasetConstraintsOrBuilder - Description: Dataset-level constraints. This is currently used for specifying information about changes in num_examples. - Protobuf Definition: optional .tensorflow.metadata.v0.DatasetConstraints dataset_constraints = 11; ``` -------------------------------- ### SparseTopKCategoricalAccuracy Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/SparseTopKCategoricalAccuracy Provides documentation for various methods within the SparseTopKCategoricalAccuracy class, including initialization checks, type checking, builder creation, and field parsing. ```APIDOC SparseTopKCategoricalAccuracy: isInitialized(): Boolean - Checks if the message has been initialized. - Returns: Boolean isInstanceOf[T0]: Boolean - Checks if the object is an instance of a specific type T0. - Returns: Boolean makeExtensionsImmutable(): Unit - Makes extensions immutable. This method is typically called internally. - Returns: Unit ne(arg0: AnyRef): Boolean - Compares this object with another for inequality. - Parameters: - arg0: The object to compare with. - Returns: Boolean indicating inequality. newBuilderForType(parent: com.google.protobuf.GeneratedMessageV3.BuilderParent): org.tensorflow.metadata.v0.SparseTopKCategoricalAccuracy.Builder - Creates a new builder for a message of the same type, associated with a parent builder. - Parameters: - parent: The parent builder. - Returns: A new Builder instance. newBuilderForType(): org.tensorflow.metadata.v0.SparseTopKCategoricalAccuracy.Builder - Creates a new builder for a message of the same type. - Returns: A new Builder instance. newBuilderForType(parent: com.google.protobuf.AbstractMessage.BuilderParent): com.google.protobuf.Message.Builder - Creates a new builder for a message of the same type, associated with a parent builder. - Parameters: - parent: The parent builder. - Returns: A new Message.Builder instance. newInstance(unused: com.google.protobuf.GeneratedMessageV3.UnusedPrivateParameter): Object - Creates a new instance of the message, typically for internal use. - Parameters: - unused: An unused parameter, required by the signature. - Returns: A new instance of the message. notify(): Unit - Wakes up a single thread that is waiting on this object's monitor. - Returns: Unit notifyAll(): Unit - Wakes up all threads that are waiting on this object's monitor. - Returns: Unit parseUnknownField(input: com.google.protobuf.CodedInputStream, unknownFields: com.google.protobuf.UnknownFieldSet.Builder, extensionRegistry: com.google.protobuf.ExtensionRegistryLite, tag: Int): Boolean - Parses an unknown field from the input stream. - Parameters: - input: The CodedInputStream to read from. - unknownFields: The builder for unknown fields. - extensionRegistry: The extension registry. - tag: The tag of the field being parsed. - Returns: Boolean indicating if the field was parsed successfully. ``` -------------------------------- ### IntDomain Class Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/IntDomain Provides documentation for various methods of the IntDomain class, including Protobuf-specific operations and standard object methods. ```APIDOC IntDomain Class Methods: isInstanceOf[T0]: Boolean - Checks if an instance is of a specific type. - Parameters: - T0: The type parameter to check against. - Returns: True if the instance is of type T0, false otherwise. makeExtensionsImmutable(): Unit - Marks extensions as immutable. This is a Protobuf internal method. - Returns: Unit. ne(arg0: AnyRef): Boolean - Checks for inequality with another object. - Parameters: - arg0: The object to compare with. - Returns: True if the objects are not equal, false otherwise. newBuilderForType(parent: com.google.protobuf.GeneratedMessageV3.BuilderParent): org.tensorflow.metadata.v0.IntDomain.Builder - Creates a new builder for this message type, associated with a parent builder. - Parameters: - parent: The parent builder. - Returns: A new IntDomain.Builder instance. newBuilderForType(): org.tensorflow.metadata.v0.IntDomain.Builder - Creates a new builder for this message type. - Returns: A new IntDomain.Builder instance. newBuilderForType(parent: com.google.protobuf.AbstractMessage.BuilderParent): com.google.protobuf.Message.Builder - Creates a new builder for this message type, accepting an AbstractMessage.BuilderParent. - Parameters: - parent: The parent builder. - Returns: A new Message.Builder instance. newInstance(unused: com.google.protobuf.GeneratedMessageV3.UnusedPrivateParameter): AnyRef - Creates a new instance of the message, typically used internally by Protobuf. - Parameters: - unused: An unused parameter required by the Protobuf API. - Returns: A new instance of the message. notify(): Unit - Wakes up a single thread that is waiting on this object's monitor. - Returns: Unit. notifyAll(): Unit - Wakes up all threads that are waiting on this object's monitor. - Returns: Unit. parseUnknownField(input: com.google.protobuf.CodedInputStream, unknownFields: com.google.protobuf.UnknownFieldSet.Builder, extensionRegistry: com.google.protobuf.ExtensionRegistryLite, tag: Int): Boolean - Parses an unknown field from the input stream. - Parameters: - input: The CodedInputStream to read from. - unknownFields: The builder for unknown fields. - extensionRegistry: The extension registry. - tag: The tag of the unknown field. - Returns: True if the field was parsed successfully, false otherwise. - Throws: java.io.IOException if an I/O error occurs. parseUnknownFieldProto3(input: com.google.protobuf.CodedInputStream, unknownFields: com.google.protobuf.UnknownFieldSet.Builder, extensionRegistry: com.google.protobuf.ExtensionRegistryLite, tag: Int): Boolean - Parses an unknown field from the input stream using Proto3 rules. - Parameters: - input: The CodedInputStream to read from. - unknownFields: The builder for unknown fields. - extensionRegistry: The extension registry. - tag: The tag of the unknown field. - Returns: True if the field was parsed successfully, false otherwise. - Throws: java.io.IOException if an I/O error occurs. ``` -------------------------------- ### FeaturePresence min_fraction Accessors Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/FeaturePresence Provides methods to get the minimum fraction of examples that have a feature and to check if this value is set. Corresponds to the 'min_fraction' field in the protobuf definition. ```Scala def getMinFraction(): Double // Minimum fraction of examples that have this feature. def hasMinFraction(): Boolean // Whether the minFraction field is set. // optional double min_fraction = 1; ``` ```APIDOC FeaturePresence: getMinFraction(): Double // Minimum fraction of examples that have this feature. // Returns: The minFraction. hasMinFraction(): Boolean // Whether the minFraction field is set. // optional double min_fraction = 1; ``` -------------------------------- ### SliceSqlOrBuilder Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/StringDomain Documentation for the SliceSqlOrBuilder interface, used for building SliceSql Protobuf messages. ```APIDOC SliceSqlOrBuilder: Description: Interface for building SliceSql Protobuf messages. ``` -------------------------------- ### FeaturePresence min_count Accessors Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/FeaturePresence Provides methods to get the minimum number of examples that have a feature and to check if this value is set. Corresponds to the 'min_count' field in the protobuf definition. ```Scala def getMinCount(): Long // Minimum number of examples that have this feature. def hasMinCount(): Boolean // Whether the minCount field is set. // optional int64 min_count = 2; ``` ```APIDOC FeaturePresence: getMinCount(): Long // Minimum number of examples that have this feature. // Returns: The minCount. hasMinCount(): Boolean // Whether the minCount field is set. // optional int64 min_count = 2; ``` -------------------------------- ### Multi-label Classification Task Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/MultiClassClassification Configuration for a multi-label classification task. In this setup, each example can be associated with zero or more labels from a set of possible classes. The model outputs probabilities for each class. ```APIDOC MultiLabelClassification: Description: Configuration for a multi-label classification task. Task Details: Predicts class probabilities for each example, where an example can belong to multiple classes. Output: A vector of probabilities, with dimension equal to the number of unique classes (n_classes) if specified, or dynamically inferred. ``` -------------------------------- ### StringDomain Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/StringDomain Provides documentation for core methods of the StringDomain class, including writing to an output stream and internal helper methods. It also highlights deprecated methods. ```APIDOC StringDomain: writeTo(output: OutputStream): Unit - Writes the StringDomain object to the specified output stream. - Parameters: - output: The OutputStream to write to. - Returns: Unit. - Annotations: @Override() - Definition Classes: AbstractMessageLite, MessageLite internalGetMapField(fieldNumber: Int): MapField[_, _] - Retrieves an internal map field by its field number. This method is deprecated. - Parameters: - fieldNumber: The integer identifier for the map field. - Returns: A MapField containing key-value pairs. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 - Annotations: @Deprecated mergeFromAndMakeImmutableInternal(input: CodedInputStream, extensionRegistry: ExtensionRegistryLite): Unit - Merges data from a CodedInputStream and makes the object immutable. This method is deprecated. - Parameters: - input: The CodedInputStream to read data from. - extensionRegistry: The ExtensionRegistryLite to handle extensions. - Returns: Unit. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 - Annotations: @throws(classOf[com.google.protobuf.InvalidProtocolBufferException]), @Deprecated ``` -------------------------------- ### Scio Custom BinaryFileReader Source: https://spotify.github.io/scio/examples/BinaryInOut.scala This Scala example demonstrates a custom `BinaryFileReader` implementation for Scio. It defines how to read binary files with a specific format, including a magic number header, record size prefix, record data, and a suffix. It overrides `start`, `readRecord`, and `end` methods to handle file reading logic. ```scala case object MyBinaryFileReader extends BinaryFileReader { This example has no state, but a record header could for example contain a number of expected records in the entire file or some block, and the state could then be used track the number of records read and to determine when the reader would need to switch modes from reading records to reading block metadata or the file footer. override type State = Unit private def fail(msg: String): Nothing = throw new IllegalStateException(msg) override def start(is: InputStream): State = { Read the expected magic number from the first bytes of the file, and fail if it is not found. val b = new Array[Byte](3) val readBytes = is.read(b) if (readBytes != b.length) fail("Failed to read header") val magicNumberOk = b(0) == 1 && b(1) == 2 && b(2) == 3 if (!magicNumberOk) fail("Failed to find correct magic number") () } override def readRecord(state: State, is: InputStream): (State, Array[Byte]) = { Read the number of expected bytes for a record, corresponding to the `framePrefix` argument. val sizeBuf = new Array[Byte](4) val sizeBytesRead = is.read(sizeBuf) If the entire file has been consumed return `null` to indicate that the read is complete. If a record count was maintained in `state`, then once all records were read a similar value should be returned. if (sizeBytesRead == -1) (state, null) else { if (sizeBytesRead != 4) fail(s"Failed to read record size $sizeBytesRead") else { val size = ByteBuffer.wrap(sizeBuf).getInt if (size < 0) fail(s"Bad record size $size") val elementBytes = new Array[Byte](size) val bytesRead = is.read(elementBytes) Ensure all expected bytes are read. if (bytesRead != size) fail("Failed to read expected record bytes") else { val suffix = is.read() Ensure `frameSuffix` is read. if (suffix != 0) fail(s"Failed to read expected record suffix $suffix") Return the record and state. If a record count was maintained in `state` it would be incremented here. else (state, elementBytes) } } } } There is no footer to read and no validation which needs to occur, so return Unit override def end(state: State, is: InputStream): Unit = () } ``` -------------------------------- ### PresenceAndValencyStatistics Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/PresenceAndValencyStatistics Provides documentation for various methods within the PresenceAndValencyStatistics class. This includes methods for creating builder instances, generating new instances, handling unknown fields, and managing object synchronization. ```APIDOC newBuilderForType(parent: com.google.protobuf.GeneratedMessageV3.BuilderParent): org.tensorflow.metadata.v0.PresenceAndValencyStatistics.Builder - Creates a new builder for the PresenceAndValencyStatistics message type, associated with a parent builder. - Attributes: protected[[v0]] - Definition Classes: PresenceAndValencyStatistics → GeneratedMessageV3 - Annotations: @Override() newBuilderForType(): org.tensorflow.metadata.v0.PresenceAndValencyStatistics.Builder - Creates a new builder for the PresenceAndValencyStatistics message type. - Definition Classes: PresenceAndValencyStatistics → Message → MessageLite - Annotations: @Override() newBuilderForType(parent: com.google.protobuf.AbstractMessage.BuilderParent): com.google.protobuf.Message.Builder - Creates a new builder for the message type with the given parent builder. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 → AbstractMessage newInstance(unused: com.google.protobuf.GeneratedMessageV3.UnusedPrivateParameter): AnyRef - Creates a new instance of the PresenceAndValencyStatistics message. - Attributes: protected[[v0]] - Definition Classes: PresenceAndValencyStatistics → GeneratedMessageV3 - Annotations: @Override() @SuppressWarnings() notify(): Unit - Notifies a single waiting thread. - Definition Classes: AnyRef - Annotations: @native() notifyAll(): Unit - Notifies all threads waiting on this object's monitor. - Definition Classes: AnyRef - Annotations: @native() parseUnknownField(input: com.google.protobuf.CodedInputStream, unknownFields: com.google.protobuf.UnknownFieldSet.Builder, extensionRegistry: com.google.protobuf.ExtensionRegistryLite, tag: Int): Boolean - Parses an unknown field from the input stream. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 - Annotations: @throws(classOf[java.io.IOException]) parseUnknownFieldProto3(input: com.google.protobuf.CodedInputStream, unknownFields: com.google.protobuf.UnknownFieldSet.Builder, extensionRegistry: com.google.protobuf.ExtensionRegistryLite, tag: Int): Boolean - Parses an unknown field from the input stream using Proto3 rules. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 - Annotations: @throws(classOf[java.io.IOException]) synchronized[T0](x$1: =>T0): T0 - Executes the given block of code synchronized on this object. - Definition Classes: AnyRef toBuilder(): org.tensorflow.metadata.v0.PresenceAndValencyStatistics.Builder - Returns a builder containing the contents of this message. - Definition Classes: PresenceAndValencyStatistics → Message → MessageLite - Annotations: @Override() ``` -------------------------------- ### Filter Dynamic Counters Source: https://spotify.github.io/scio/examples/MetricsExample.scala Illustrates how to retrieve and filter dynamic metrics, specifically counters whose names start with 'even_'. It iterates through the filtered counters and prints their committed values. ```Scala result.allCounters.view .filterKeys(_.getName.startsWith("even_")) .foreach { case (name, value) => println(name.getName + ": " + value.committed.get) } ``` -------------------------------- ### Image Domain Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/ImageDomain Documentation for the ImageDomain component. ```APIDOC ImageDomain[**](../../../../org/tensorflow/metadata/v0/ImageDomain.html "Permalink") ### ``` -------------------------------- ### SequenceLengthConstraints Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/SequenceLengthConstraints Provides documentation for various methods of the SequenceLengthConstraints class, covering initialization, state checking, and internal operations. Includes details on parameters, return types, and definition classes. ```APIDOC SequenceLengthConstraints: hasMinSequenceLength(): Boolean - Returns whether the minSequenceLength field is set. - Related field: `optional int64 min_sequence_length = 3;` - Definition Classes: SequenceLengthConstraints → SequenceLengthConstraintsOrBuilder hasOneof(oneof: OneofDescriptor): Boolean - Checks if a oneof field is set. - Parameters: - oneof: The OneofDescriptor to check. - Returns: Boolean indicating if the oneof is set. - Definition Classes: GeneratedMessageV3 → AbstractMessage → MessageOrBuilder hashCode(): Int - Computes the hash code for the object. - Returns: The hash code. - Definition Classes: SequenceLengthConstraints → AbstractMessage → Message → AnyRef → Any internalGetFieldAccessorTable(): GeneratedMessageV3.FieldAccessorTable - Retrieves the field accessor table for internal use. - Returns: The FieldAccessorTable. - Attributes: protected[[v0](index.html)] - Definition Classes: SequenceLengthConstraints → GeneratedMessageV3 internalGetMapFieldReflection(fieldNumber: Int): MapFieldReflectionAccessor - Retrieves map field reflection accessor for a given field number. - Parameters: - fieldNumber: The number of the field. - Returns: The MapFieldReflectionAccessor. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 isInitialized(): Boolean - Checks if all required fields are set. - Returns: True if initialized, false otherwise. - Definition Classes: SequenceLengthConstraints → GeneratedMessageV3 → AbstractMessage → MessageLiteOrBuilder isInstanceOf[T0]: Boolean - Checks if the object is an instance of a specific type. - Parameters: - T0: The type to check against. - Returns: True if the object is an instance of T0. - Definition Classes: Any makeExtensionsImmutable(): Unit - Makes extensions immutable, typically called during initialization. - Definition Classes: GeneratedMessageV3 ne(arg0: AnyRef): Boolean - Compares this object with another for inequality. - Parameters: - arg0: The object to compare with. - Returns: True if the objects are not equal. - Definition Classes: AnyRef newBuilderForType(parent: BuilderParent): SequenceLengthConstraints.Builder - Creates a new builder for this message type with a parent builder. - Parameters: - parent: The parent builder. - Returns: A new Builder instance. - Attributes: protected[[v0](index.html)] - Definition Classes: SequenceLengthConstraints → GeneratedMessageV3 newBuilderForType(): SequenceLengthConstraints.Builder - Creates a new builder for this message type. - Returns: A new Builder instance. - Definition Classes: SequenceLengthConstraints → Message → MessageLite newBuilderForType(parent: AbstractMessage.BuilderParent): Message.Builder - Creates a new builder for this message type with an abstract message parent. - Parameters: - parent: The parent builder. - Returns: A new Message.Builder instance. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 → AbstractMessage ``` -------------------------------- ### Multi-Class Classification Task Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/LogisticRegression Configuration for a multi-class classification problem. In this setup, a model predicts exactly one label from a predefined set of 'n_classes' possible labels for each input example. ```APIDOC MultiClassClassification: Description: Defines parameters for a multi-class classification task. Characteristics: - Number of classes: 'n_classes' specifies the total count of distinct possible labels. - Prediction: The model outputs a single label per example. - Output Type: Corresponds to the data type of the label column. ``` -------------------------------- ### Multi-Class Classification Task Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/LifecycleStage Configuration for a multi-class classification problem. In this setup, a model predicts exactly one label from a predefined set of 'n_classes' possible labels for each input example. ```APIDOC MultiClassClassification: Description: Defines parameters for a multi-class classification task. Characteristics: - Number of classes: 'n_classes' specifies the total count of distinct possible labels. - Prediction: The model outputs a single label per example. - Output Type: Corresponds to the data type of the label column. ``` -------------------------------- ### TimeDomain General Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/TimeDomain Provides documentation for general methods of the TimeDomain class, including checking for fields, hash code generation, and initialization status. ```scala def hasField(field: FieldDescriptor): Boolean - Checks if a specific field is set. - Definition Classes: GeneratedMessageV3, MessageOrBuilder ``` ```scala def hasOneof(oneof: OneofDescriptor): Boolean - Checks if any field in a oneof is set. - Definition Classes: GeneratedMessageV3, AbstractMessage, MessageOrBuilder ``` ```scala def hashCode(): Int - Returns the hash code value for this object. - Definition Classes: TimeDomain, AbstractMessage, Message, AnyRef, Any ``` ```scala def internalGetFieldAccessorTable(): FieldAccessorTable - Retrieves the internal field accessor table. - Attributes: protected[[v0]] - Definition Classes: TimeDomain, GeneratedMessageV3 ``` ```scala def internalGetMapFieldReflection(fieldNumber: Int): MapFieldReflectionAccessor - Gets the map field reflection accessor for a given field number. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 ``` ```scala final def isInitialized(): Boolean - Checks if all required fields are initialized. - Definition Classes: TimeDomain, GeneratedMessageV3, AbstractMessage, MessageLiteOrBuilder ``` ```scala final def isInstanceOf[T0]: Boolean - Checks if the object is an instance of a given type. - Definition Classes: Any ``` ```scala def makeExtensionsImmutable(): Unit - Makes extensions immutable. - Attributes: protected[protobuf] - Definition Classes: GeneratedMessageV3 ``` ```scala final def ne(arg0: AnyRef): Boolean - Compares this object to the specified object. - Definition Classes: AnyRef ``` ```scala def newBuilderForType(parent: BuilderParent): Builder - Creates a new builder for the same message type. - Attributes: protected[[v0]] - Definition Classes: TimeDomain, GeneratedMessageV3 ``` -------------------------------- ### Multi-Class Classification Task Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/InfinityNorm Configuration for a multi-class classification problem. In this setup, a model predicts exactly one label from a predefined set of 'n_classes' possible labels for each input example. ```APIDOC MultiClassClassification: Description: Defines parameters for a multi-class classification task. Characteristics: - Number of classes: 'n_classes' specifies the total count of distinct possible labels. - Prediction: The model outputs a single label per example. - Output Type: Corresponds to the data type of the label column. ``` -------------------------------- ### DriftSkewInfoOrBuilder API Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/DriftSkewInfoOrBuilder Provides documentation for various abstract methods within the DriftSkewInfoOrBuilder interface. These methods allow access to initialization errors, all fields, default instances, type descriptors, and detailed drift/skew measurements. ```APIDOC DriftSkewInfoOrBuilder: findInitializationErrors(): List[String] - Returns a list of initialization errors for the object. - Parameters: None - Returns: A List of Strings representing initialization errors. getAllFields(): Map[FieldDescriptor, AnyRef] - Returns a map of all fields in the message, mapping field descriptors to their values. - Parameters: None - Returns: A Map where keys are FieldDescriptor and values are AnyRef. getDefaultInstanceForType(): Message - Returns the default instance of the message for the current type. - Parameters: None - Returns: A Message object representing the default instance. getDescriptorForType(): Descriptor - Returns the descriptor for the message type. - Parameters: None - Returns: A Descriptor object. getDriftMeasurements(index: Int): Measurement - Retrieves a specific drift measurement by its index. - Parameters: - index: The zero-based index of the measurement to retrieve. - Returns: A Measurement object. - Description: The drift/skew may be measured in the same invocation of TFDV, in which case both of the following fields are populated. Also the drift/skew may be quantified by different measurements, thus repeated. getDriftMeasurementsCount(): Int - Returns the total number of drift measurements available. - Parameters: None - Returns: An Int representing the count of drift measurements. - Description: The drift/skew may be measured in the same invocation of TFDV, in which case both of the following fields are populated. Also the drift/skew may be quantified by different measurements, thus repeated. getDriftMeasurementsList(): List[Measurement] - Returns a list of all drift measurements. - Parameters: None - Returns: A List of Measurement objects. - Description: The drift/skew may be measured in the same invocation of TFDV, in which case both of the following fields are populated. Also the drift/skew may be quantified by different measurements, thus repeated. getDriftMeasurementsOrBuilder(index: Int): MeasurementOrBuilder - Retrieves an orBuilder for a specific drift measurement by its index. - Parameters: - index: The zero-based index of the measurement to retrieve. - Returns: A MeasurementOrBuilder object. - Description: The drift/skew may be measured in the same invocation of TFDV, in which case both of the following fields are populated. Also the drift/skew may be quantified by different measurements, thus repeated. ``` -------------------------------- ### SensitivityAtSpecificityOrBuilder Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/StringDomain Documentation for the SensitivityAtSpecificityOrBuilder interface, used for building SensitivityAtSpecificity metric configurations. ```APIDOC SensitivityAtSpecificityOrBuilder: Description: Interface for building SensitivityAtSpecificity metric configurations. ``` -------------------------------- ### ScioContext Execution Flow Source: https://spotify.github.io/scio/examples/TypedBigQueryTornadoes.scala Initiates the execution of a Scio pipeline and waits for its completion. The `run()` method starts the underlying execution engine, and `waitUntilDone()` blocks until the job finishes, returning a `ScioResult`. ```Scala import com.spotify.scio.ScioContext import scala.concurrent.duration._ // Assuming 'sc' is an instance of ScioContext val sc: ScioContext = ??? // Execute the pipeline and wait for completion val result = sc.run().waitUntilDone(1.hour, cancelJob = false) // The 'result' variable now holds the ScioResult, which can be inspected for job status and metrics. ``` -------------------------------- ### Retrieve Counter Values at Steps Source: https://spotify.github.io/scio/examples/MetricsExample.scala Illustrates how to retrieve metric values for a counter (e.g., 'sum2') at different steps within the pipeline. It uses `counterAtSteps` to get a map of step names to metric values. ```Scala val s2steps = result.counterAtSteps(sum2).view.mapValues(_.committed.get).toMap s2steps.foreach { case (step, value) => println(s"sum2 at $step: " + value) } ``` -------------------------------- ### IntDomain Class Methods Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/IntDomain Provides documentation for various methods available in the IntDomain class, including equality checks, cloning, and introspection methods common to Scala/Java objects and Protobuf messages. ```APIDOC IntDomain Class Methods: - !=(arg0: Any): Boolean - Description: Checks for inequality with another object. - Parameters: - arg0: The object to compare against. - Returns: True if the objects are not equal, false otherwise. - Definition Classes: AnyRef → Any - ##: Int - Description: Returns the hash code of the object. - Returns: The hash code as an Int. - Definition Classes: AnyRef → Any - ==(arg0: Any): Boolean - Description: Checks for equality with another object. - Parameters: - arg0: The object to compare against. - Returns: True if the objects are equal, false otherwise. - Definition Classes: AnyRef → Any - asInstanceOf[T0]: T0 - Description: Casts the object to a specified type T0. - Type Parameters: - T0: The target type. - Returns: The object cast to type T0. - Definition Classes: Any - clone(): AnyRef - Description: Creates and returns a copy of this object. - Attributes: protected[lang] - Definition Classes: AnyRef - Annotations: @throws(classOf[java.lang.CloneNotSupportedException]) @native() - eq(x$1: AnyRef): Boolean - Description: Tests whether two references point to the same object. - Parameters: - x$1: The reference to compare. - Returns: True if both references point to the same object, false otherwise. - Definition Classes: AnyRef - equals(obj: Object): Boolean - Description: Compares this object with the specified object for equality. - Parameters: - obj: The object to compare with. - Returns: True if the specified object is equal to this object, false otherwise. - Definition Classes: IntDomain → AbstractMessage → Message → AnyRef → Any - Annotations: @Override() - finalize(): Unit - Description: Called by the garbage collector on an object when garbage collection determines that there are no more references to the object. - Attributes: protected[lang] - Definition Classes: AnyRef - Annotations: @throws(classOf[java.lang.Throwable]) - findInitializationErrors(): java.util.List[String] - Description: Returns a list of initialization errors found in the message. - Returns: A list of strings, where each string is an initialization error. - Definition Classes: AbstractMessage → MessageOrBuilder - getAllFields(): java.util.Map[com.google.protobuf.Descriptors.FieldDescriptor,Object] - Description: Returns a map containing all fields of the message and their values. - Returns: A map where keys are FieldDescriptors and values are the field values. - Definition Classes: GeneratedMessageV3 → MessageOrBuilder - getClass(): Class[_ <: AnyRef] - Description: Returns the runtime class of this object. - Returns: The class of this object. - Definition Classes: AnyRef → Any - Annotations: @native() - getDefaultInstanceForType(): org.tensorflow.metadata.v0.IntDomain - Description: Returns the default instance of the message for the specified type. - Returns: The default instance of IntDomain. - Definition Classes: IntDomain → MessageOrBuilder → MessageLiteOrBuilder - Annotations: @Override() - getDescriptorForType(): com.google.protobuf.Descriptors.Descriptor - Description: Returns the descriptor for the message type. - Returns: The Descriptor for the IntDomain type. - Definition Classes: GeneratedMessageV3 → MessageOrBuilder - getField(field: com.google.protobuf.Descriptors.FieldDescriptor): Object - Description: Returns the value of the specified field. - Parameters: - field: The descriptor of the field to retrieve. - Returns: The value of the specified field. - Definition Classes: GeneratedMessageV3 → MessageOrBuilder - getInitializationErrorString(): String - Description: Returns a string representation of any initialization errors. - Returns: A string describing initialization errors, or null if none exist. - Definition Classes: AbstractMessage → MessageOrBuilder ``` -------------------------------- ### SparseFeatureOrBuilder Documentation Source: https://spotify.github.io/scio/api/org/tensorflow/metadata/v0/StringDomain Documentation for the SparseFeatureOrBuilder interface, used for building SparseFeature Protobuf messages. ```APIDOC SparseFeatureOrBuilder: Description: Interface for building SparseFeature Protobuf messages. ```