### Visualizing Continuous Data with Partition and Series in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example shows how to use `Partition` with a `Series` of `Mean` and `Extrema` stats to summarize continuous data. It reuses the `y` data from the previous example, fits the `Partition` object to calculate mean and extrema for sections of the data stream, and then plots these summaries, providing insights into central tendency and range over time. ```Julia o = Partition(Series(Mean(), Extrema())) fit!(o, y) plot(o) ``` -------------------------------- ### Creating and Fitting a Series in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/collections.md This example demonstrates how to create a `Series` object to track multiple statistics (Mean and Variance) on a single data stream. It then fits the `Series` with a randomly generated array `y`. ```Julia y = rand(1000) s = Series(Mean(), Variance()) fit!(s, y) ``` -------------------------------- ### Demonstrating OnlineStat Merging (Sequential Example) Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/bigdata.md This snippet illustrates the `merge!` operation for `OnlineStat` objects. It initializes three `Series` objects with `Mean`, `Variance`, and `KHist` statistics, fits them to separate random datasets, and then sequentially merges the results into a single `Series` object, demonstrating how partial statistics can be combined. ```Julia y1 = randn(10_000) y2 = randn(10_000) y3 = randn(10_000) a = Series(Mean(), Variance(), KHist(20)) b = Series(Mean(), Variance(), KHist(20)) c = Series(Mean(), Variance(), KHist(20)) fit!(a, y1) fit!(b, y2) fit!(c, y3) merge!(a, b) # merge `b` into `a` merge!(a, c) # merge `c` into `a` ``` -------------------------------- ### Quickstart: Initializing and Updating Online Statistics in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/README.md This snippet demonstrates the basic workflow for using OnlineStats.jl. It covers adding the package, creating a Series of online statistics (Mean, Variance, Extrema), updating these statistics incrementally with single data points or large arrays, and retrieving their current computed values. This showcases the library's efficiency for processing data streams. ```Julia import Pkg Pkg.add("OnlineStats") using OnlineStats # Create several statistics o = Series(Mean(), Variance(), Extrema()) # Update with single data point fit!(o, 1.0) # Iterate through and update with lots of data fit!(o, randn(10^6)) # Get the values of the statistics value(o) # (value(mean), value(variance), value(extrema)) ``` -------------------------------- ### Environment Setup for OnlineStats.jl Documentation Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/stats_and_models.md This snippet sets up environment variables for GKS (Graphics Kernel System) within a Julia documentation build process. It ensures that plots are rendered correctly and character encoding is set to UTF-8, which is crucial for displaying various symbols and text in plots. ```Julia ENV["GKSwstype"] = "100" ENV["GKS_ENCODING"]="utf8" ``` -------------------------------- ### Setting up Plotting Environment for OnlineStats.jl Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/bigdata.md This snippet configures environment variables for GKS (Graphics Kernel System) to ensure plots are generated correctly, specifically setting the workstation type and encoding for `Plots.jl` when used with `OnlineStats.jl` in a documentation setup. ```Julia ENV["GKSwstype"] = "100" ENV["GKS_ENCODING"]="utf8" ``` -------------------------------- ### Generating a Bivariate HeatMap in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example shows how to create a `HeatMap` for bivariate continuous data. It initializes a `HeatMap` with specified x and y ranges, generates one million correlated data points, fits the `HeatMap` with zipped `(x, y)` pairs, and then plots the resulting density map, effectively visualizing the joint distribution of two variables. ```Julia o = HeatMap(-5:.1:5, -0:.1:10) x, y = randn(10^6), 5 .+ randn(10^6) fit!(o, zip(x, y)) plot(o) ``` -------------------------------- ### Implementing the Mean OnlineStat Type (Julia) Source: https://github.com/joshday/onlinestats.jl/blob/master/paper/paper.md This comprehensive example demonstrates how to implement a new `OnlineStat` type, `Mean`, in Julia. It includes the `mutable struct` definition, a constructor, and the `_fit!` and `_merge!` methods, showing how to translate mathematical update and merge formulas into functional Julia code for online mean calculation with a weighting function. ```Julia mutable struct Mean{T,W} <: OnlineStat{Number} m::T weight::W n::Int end function Mean(T::Type{<:Number} = Float64; weight = inv) Mean(zero(T), weight, 0) end function _fit!(o::Mean{T}, x) where {T} o.n += 1 w = T(o.weight(o.n)) o.m += w * (x - o.m) end function _merge!(o::Mean, o2::Mean) o.n += o2.n o.m += (o2.n / o.n) * (o2.m - o.m) end ``` -------------------------------- ### Tracking OnlineStat Evolution with Trace in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example illustrates using the `Trace` type to monitor how an `OnlineStat` changes over time as new observations are added. It generates a time-varying dataset, initializes a `Trace` object to track the `Extrema` of the data, fits it with one million data points, and then plots the trace, which can reveal concept drift or aid in hyperparameter tuning. ```Julia y = range(1, 20, length=10^6) .* randn(10^6) o = Trace(Extrema()) fit!(o, y) plot(o) ``` -------------------------------- ### Visualizing Bivariate Continuous Data with IndexedPartition and KHist in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example demonstrates plotting one continuous variable against another using `IndexedPartition`. It generates two correlated datasets (`x` and `y`), fits an `IndexedPartition` (configured for `Float64` x-axis and `KHist(40)` for y-axis summaries) with zipped `(x, y)` pairs, and then plots the resulting bivariate summary, showing the relationship between `x` and `y`. ```Julia x = randn(10^6) y = x + randn(10^6) o = fit!(IndexedPartition(Float64, KHist(40), 40), zip(x, y)) plot(o) ``` -------------------------------- ### Creating and Fitting a Group in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/collections.md This example illustrates how to create a `Group` object to track statistics (Mean and CountMap for Booleans) on different data streams. It fits the `Group` with an iterator `itr` combining a random normal array and a random boolean array. ```Julia g = Group(Mean(), CountMap(Bool)) itr = zip(randn(100), rand(Bool, 100)) fit!(g, itr) ``` -------------------------------- ### Fitting CovMatrix with Matrix Rows/Columns in OnlineStats.jl Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/howfitworks.md This example demonstrates how to fit a `CovMatrix` in OnlineStats.jl using a 2D `Matrix`. It clarifies the ambiguity of matrix input by explicitly using `eachrow` and `eachcol` to treat rows or columns as individual observations, respectively. This ensures correct statistical updates for multivariate data. ```Julia x = randn(1000, 2) fit!(CovMatrix(), eachrow(x)) fit!(CovMatrix(), eachcol(x')) ``` -------------------------------- ### Processing CSV Data with OnlineStats.jl for Grouped Histograms Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/bigdata.md This example demonstrates reading a large CSV file efficiently using `CSV.Rows` and `download` to fetch data. It then processes the data incrementally with `OnlineStats.GroupBy` and `Hist` to calculate grouped histograms of 'sepal_length' by 'variety', finally visualizing the results. ```Julia using OnlineStats, CSV, Plots url = "https://gist.githubusercontent.com/joshday/df7bdaa1d58b398592e7656395de6335/raw/5a1c83f498f8ca7e25ff2372340e44b3389be9b1/iris.csv" rows = CSV.Rows(download(url); reusebuffer = true) itr = (string(row.variety) => parse(Float64, row.sepal_length) for row in rows) o = GroupBy(String, Hist(4:0.25:8)) fit!(o, itr) plot(o, layout=(3,1)) ``` -------------------------------- ### Approximating Cumulative Distribution Function with OrderStats in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example illustrates how to approximate the Cumulative Distribution Function (CDF) using the `OrderStats` type. It fits an `OrderStats` object, which maintains a fixed number of order statistics, to one million random data points, and then plots the approximate CDF, useful for understanding the distribution of data without storing all observations. ```Julia o = fit!(OrderStats(1000), randn(10^6)) plot(o) ``` -------------------------------- ### Visualizing Time-Series Data with IndexedPartition and Date Formatting in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example addresses plotting `Date` or `DateTime` types with `IndexedPartition` and ensuring human-readable tick labels. It generates date-based `x` data and continuous `y` data, fits an `IndexedPartition` (configured for `Date` x-axis and `KHist(20)` for y-axis summaries), and then plots the result using an `xformatter` to convert numeric date values back to formatted date strings for clarity. ```Julia using Dates x = rand(Date(2019):Day(1):Date(2020), 10^6) y = Dates.value.(x) .+ 30randn(10^6) o = fit!(IndexedPartition(Date, KHist(20)), zip(x,y)) plot(o, xformatter = x -> string(Date(Dates.UTInstant(Day(x))))) ``` -------------------------------- ### Comparing Multiple Histogram Types with Series in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This example showcases different histogram types (`KHist`, `Hist`, `ExpandingHist`) combined within a `Series` object. It generates one million random data points, fits them to the `Series` containing these histograms, and then plots all three histograms on a single graph with linked x-axes and custom labels, allowing for direct comparison of their binning and representation methods. ```Julia s = fit!(Series(KHist(25), Hist(-5:.2:5), ExpandingHist(100)), randn(10^6)) plot(s, link = :x, label = ["KHist" "Hist" "ExpandingHist"]) ``` -------------------------------- ### Performing Distributed Parallel Computation with OnlineStats.jl Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/bigdata.md This example demonstrates distributed parallel computation using Julia's `Distributed` module. It adds worker processes, loads `OnlineStats` on all workers, and then uses `@distributed merge` to fit `Series` objects to separate datasets in parallel, finally merging the results into a single `OnlineStat` object `s`. ```Julia using Distributed addprocs(3) @everywhere using OnlineStats s = @distributed merge for i in 1:3 o = Series(Mean(), Variance(), KHist(20)) fit!(o, randn(10_000)) end ``` -------------------------------- ### Demonstrating Common Error: Fitting Mean with Invalid String Input Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/howfitworks.md This example highlights a common error when using `fit!` in OnlineStats.jl: attempting to fit a `Mean` statistic with an incompatible data type, such as a `String`. Since `Mean` expects `Number` observations, the system attempts to iterate the string, leading to an error when it encounters characters instead of numbers. ```Julia fit!(Mean(), "asdf") ``` -------------------------------- ### Comparing Weighting in StatsBase.jl and OnlineStats.jl (Julia) Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/weights.md This example highlights the fundamental difference in how `StatsBase.jl` and `OnlineStats.jl` handle observation weights. `StatsBase` applies weights to the overall calculation, while `OnlineStats` uses weights to determine the influence of new observations relative to the current state of the statistic. It shows that `mean(x)` in `StatsBase` with uniform weights is equivalent to unweighted mean, whereas `OnlineStats` with a constant weight (e.g., 0.1) gives higher influence to recent observations. ```Julia using OnlineStats, StatsBase x = 1:99; w = fill(0.1, 99); # StatsBase: All weights == 0.1 mean(x) ≈ mean(x, aweights(w)) ≈ mean(x, fweights(w)) ≈ mean(x, pweights(w)) # OnlineStats: All weights == 0.1 o = fit!(Mean(weight = n -> 0.1), x) mean(x) # Every observation has equal influence over statistic. value(o) # Recent observations have higher influence over statistic. ``` -------------------------------- ### Customizing HeatMap Plot with Marginals and Legend in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This snippet demonstrates customizing the `HeatMap` plot generated in the previous example. It re-plots the existing `HeatMap` object `o`, explicitly disabling marginal histograms (`marginals=false`) and enabling the legend (`legend=true`), allowing for a focused view of the joint distribution without the individual variable distributions. ```Julia plot(o, marginals=false, legend=true) ``` -------------------------------- ### Fitting StatLearn Models with SGD and MSPI in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/ml.md This snippet demonstrates how to initialize and fit `StatLearn` models using two different algorithms, Stochastic Gradient Descent (SGD) and Majorization-Minimization (MSPI), with a specified learning rate. It then plots the evolution of coefficients over time using `Trace` to visualize the learning process. ```Julia using OnlineStats, Plots # fake data x = rand(Bool, 1000, 10) y = x * (1:10) + 10randn(1000) rate = LearningRate(.8) o = Trace(StatLearn(SGD(), OnlineStats.l2regloss; rate)) o2 = Trace(StatLearn(MSPI(), OnlineStats.l2regloss; rate)) itr = zip(eachrow(x), y) fit!(o, itr) fit!(o2, itr) plot( plot(o, xlab="Nobs", title="SGD Coefficients", lab=nothing), plot(o2, xlab="Nobs", title="MSPI Coefficients", lab=nothing), link=:y ) ``` -------------------------------- ### Visualizing Continuous Data with Partition and KHist in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This snippet demonstrates how to visualize a large stream of continuous data using `Partition` with a `KHist` (K-Histogram) in OnlineStats.jl. It generates one million data points, fits them to a `Partition` object configured with `KHist(10)` for binning, and then plots the summarized histogram, efficiently representing the data distribution. ```Julia y = cumsum(randn(10^6)) + 100randn(10^6) o = Partition(KHist(10)) fit!(o, y) plot(o) ``` -------------------------------- ### Visualizing Categorical Data with Partition and CountMap in Julia Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/dataviz.md This snippet illustrates visualizing categorical data using `Partition` with `CountMap`. It generates one million random categorical values, fits them to a `Partition` configured with `CountMap(String)` to count occurrences of each category, and then plots the resulting summary, effectively showing the distribution of categories across partitions. ```Julia y = rand(["a", "a", "b", "c"], 10^6) o = Partition(CountMap(String), 75) fit!(o, y) plot(o) ``` -------------------------------- ### GitHub Star Button HTML Integration Source: https://github.com/joshday/onlinestats.jl/blob/master/docs/src/index.md This HTML snippet displays a GitHub 'Star' button for the `joshday/OnlineStats.jl` repository. It encourages users to star the project and includes the necessary script to render the button. ```HTML