### Define Datasets for Examples Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Defines two sample datasets, ds1 and ds2, used in subsequent join and concatenation examples. ```clojure (def ds1 (tc/dataset {:a [1 2 1 2 3 4 nil nil 4] :b (range 101 110) :c (map str "abs tract")})) (def ds2 (tc/dataset {:a [nil 1 2 5 4 3 2 1 nil] :b (range 110 101 -1) :c (map str "datatable") :d (symbol "X") :e [3 4 5 6 7 nil 8 1 1]})) ds1 ``` -------------------------------- ### Create a data.table in R Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates how to create a data.table in R. Ensure the `data.table` and `knitr` packages are installed. ```R DT = data.table( ID = c("b","b","b","a","a","c"), a = 1:6, b = 7:12, c = 13:18 ) kable(DT) ``` ```R class(DT$ID) ``` -------------------------------- ### Install Tablecloth and Dependencies Source: https://context7.com/scicloj/tablecloth/llms.txt Add Tablecloth to your project's dependencies and require necessary namespaces for use. ```clojure ;; deps.edn {:deps {scicloj/tablecloth {:mvn/version "8.0.16"}}} ;; Require in your namespace (require '[tablecloth.api :as tc] '[tablecloth.column.api :as tcc] '[tech.v3.datatype.functional :as dfn]) ``` -------------------------------- ### Tidyr Example: Join Columns with Missing Substitution and No Drop Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates joining columns from a dataset, similar to tidyr's `unite` function. This example keeps source columns and substitutes missing values with 'NA'. ```clojure (def df (tc/dataset {:x ["a" "a" nil nil] :y ["b" nil "b" nil]})) (tc/join-columns df "z" [:x :y] {:drop-columns? false :missing-subst "NA" :separator "_"})) ``` -------------------------------- ### Tidyr Example: Join Columns with Separator and No Drop Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Another tidyr-like example for joining columns. This version uses a specified separator and does not drop the original columns, handling `nil` values by including them directly in the joined string. ```clojure (def df (tc/dataset {:x ["a" "a" nil nil] :y ["b" nil "b" nil]})) (tc/join-columns df "z" [:x :y] {:drop-columns? false :separator "_"})) ``` -------------------------------- ### Get Tablecloth Version Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the version of the tablecloth library from the project.clj file. ```clojure (def tablecloth-version (nth (read-string (slurp "project.clj")) 2)) ``` -------------------------------- ### Selecting and Dropping Columns Source: https://context7.com/scicloj/tablecloth/llms.txt Provides examples for creating new datasets by selecting specific columns or dropping unwanted ones. This functionality works on both regular and grouped datasets. ```clojure ;; Select float64 columns (tc/select-columns DS :type/float64) ;; Select all but :V1 (tc/select-columns DS (complement #{:V1})) ;; Select specific columns (tc/select-columns DS [:V1 :V2]) ;; Drop float64 columns (tc/drop-columns DS :type/float64) ;; Drop all except :V1 and :V2 (tc/drop-columns DS (complement #{:V1 :V2})) ;; Works on grouped datasets too (-> DS (tc/group-by :V1) (tc/select-columns [:V2 :V3]) (tc/groups->map)) ``` -------------------------------- ### Select float64 columns by datatype Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Select column names based on their datatype. This example specifically targets columns with the `:float64` type. ```clojure (tc/column-names DS #{:float64} :datatype) ``` ```clojure (tc/column-names DS :type/float64) ``` -------------------------------- ### Require Tablecloth API Source: https://github.com/scicloj/tablecloth/blob/master/README.md Import the Tablecloth API for use in your Clojure project. This is a common setup step. ```clojure (require '[tablecloth.api :as tc]) ``` -------------------------------- ### Get Groups as Sequence from Grouped Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates how to obtain the groups from a dataset that has already been grouped, as a sequence of datasets. This uses a temporary dataset for illustration. ```clojure (let [ds (-> {"a" [1 1 2 2] "b" ["a" "b" "c" "d"]} (tc/dataset) (tc/group-by "a"))] (seq (ds :data))) ;; seq is not necessary but Markdown treats `:data` as command here ``` -------------------------------- ### Set Multiple Key/Index Orders Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Sets the order of multiple columns to be used as keys or indices. This example orders by :V4 then :V1. ```clojure (tc/order-by DS [:V4 :V1]) ``` -------------------------------- ### Select integer columns by datatype regex Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Select column names based on a regex matching their datatype. This example retrieves all integer columns. ```clojure (tc/column-names DS #"^:int.*" :datatype) ``` ```clojure (tc/column-names DS :type/integer) ``` -------------------------------- ### Require Namespaces and Define Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Requires the necessary namespaces for Tablecloth and tech.v3.datatype.functional, and defines a sample dataset for examples. ```clojure (require '[tablecloth.api :as tc] '[tech.v3.datatype.functional :as dfn]) ``` ```clojure (def DS (tc/dataset {:V1 (take 9 (cycle [1 2])) :V2 (range 1 10) :V3 (take 9 (cycle [0.5 1.0 1.5])) :V4 (take 9 (cycle ["A" "B" "C"]))})) ``` -------------------------------- ### Get Column Names After Grouping by :V1 Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates how to retrieve the column names of a dataset after grouping it by the :V1 column. ```clojure (-> DS (tc/group-by :V1) (tc/column-names)) ``` -------------------------------- ### Get Basic Dataset Info Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves a concise summary of the dataset, including its name, whether it's grouped, and its row and column counts. Useful for a quick overview. ```clojure (tc/info ds :basic) ``` -------------------------------- ### Get Columns as Vectors Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieve columns from a dataset as a sequence of vectors. Each vector represents a column's data. ```clojure (tc/columns DS :as-vecs) ``` -------------------------------- ### Define Datasets for Set Operations Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Defines two sample datasets, x and y, with a common column 'V1' to be used in set operation examples. ```clojure (def x (tc/dataset {:V1 [1 2 2 3 3]})) ``` ```clojure (def y (tc/dataset {:V1 [2 2 3 4 4]})) ``` -------------------------------- ### Get General Dataset Info Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Provides comprehensive information about the dataset, including column names, data types, validity counts, and basic statistics. ```clojure (tc/info ds) ``` -------------------------------- ### Get Full Dataset Information Source: https://context7.com/scicloj/tablecloth/llms.txt Obtain comprehensive information about the dataset, including column details and statistics. Options for basic or column-only info are available. ```clojure (def ds (tc/dataset "https://vega.github.io/vega-lite/examples/data/seattle-weather.csv")) ;; Full column info with statistics (tc/info ds) ;; Just name, row count, column count, grouped? (tc/info ds :basic) ;; Column metadata only (tc/info ds :columns) ``` -------------------------------- ### Get Column Names of Grouped Dataset as Regular Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Shows how to group a dataset by :V1 and then treat the grouped result as a regular dataset to inspect its column names. ```clojure (-> DS (tc/group-by :V1) (tc/as-regular-dataset) (tc/column-names)) ``` -------------------------------- ### Concatenate Multiple Datasets Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Concatenates rows from multiple datasets. This example combines `ds1` with `ds2` after dropping specific columns from `ds2`. ```clojure (tc/concat ds1 (tc/drop-columns ds2 :d)) ``` -------------------------------- ### Create and Inspect Columns - `tablecloth.column.api` Source: https://context7.com/scicloj/tablecloth/llms.txt Demonstrates creating columns from various sources (empty, vector, sequence) and inspecting their types. Requires `tablecloth.column.api` namespace. ```clojure (require '[tablecloth.column.api :as tcc]) ;; Creation (tcc/column) ;; empty column (tcc/column [1 2 3 4 5]) ;; from vector (tcc/column `(1 2 3 4 5)) ;; from sequence (tcc/ones 10) ;; column of 10 ones (tcc/zeros 10) ;; column of 10 zeros ;; Type inspection (tcc/column? [1 2 3]) ;; => false (tcc/column? (tcc/column [1 2 3])) ;; => true (tcc/typeof (tcc/column [1 2 3])) ;; => :int64 (tcc/typeof? (tcc/column [1 2 3]) :integer) ;; => true (general type) (tcc/typeof? (tcc/column [1 2 3]) :int64) ;; => true (concrete type) ``` -------------------------------- ### Import Nippy File and Take Head Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Load data from a `.nippy` file and display the first 10 rows using `tc/head`. ```clojure (-> (tc/dataset "data/example-genres.nippy") (tc/head 10)) ``` -------------------------------- ### Column Access with get and nth Source: https://github.com/scicloj/tablecloth/blob/master/docs/column_api.html Access elements within a column using the standard Clojure vector access methods `get` and `nth`. ```APIDOC ## Column Access ### Description Accesses a specific element in a column by its index. ### Methods * **get** * **nth** ### Parameters * **index** (Integer) - The zero-based index of the element to retrieve. ### Response * **Value** - The element at the specified index. ### Example ```clojure (-> (tcc/column [1 2 3 4 5]) (get 3)) ;; => 4 (-> (tcc/column [1 2 3 4 5]) (nth 3)) ;; => 4 ``` ``` -------------------------------- ### Access and Slice Columns - `tablecloth.column.api` Source: https://context7.com/scicloj/tablecloth/llms.txt Shows how to access elements by index and create sub-columns using slicing. Slicing supports various start, end, and step configurations. ```clojure (require '[tablecloth.column.api :as tcc]) ;; Access (get (tcc/column [10 20 30]) 1) ;; => 20 (nth (tcc/column [10 20 30]) 1) ;; => 20 ;; Slicing (tcc/slice (tcc/column (range 10)) 5) ;; from index 5 to end (tcc/slice (tcc/column (range 10)) 1 4) ;; indices 1..4 (tcc/slice (tcc/column (range 10)) 0 9 2) ;; every 2nd from 0..9 (tcc/slice (tcc/column (range 10)) :start :end 2) ;; every 2nd ``` -------------------------------- ### Selecting Column Names Source: https://context7.com/scicloj/tablecloth/llms.txt Illustrates how to select column names from a dataset using various selectors like keywords, sequences, regex, predicates, and type keywords. It also shows how to match on both name and datatype simultaneously. ```clojure (tc/column-names DS) ;; all names (tc/column-names DS :V1) ;; [:V1] (tc/column-names DS [:V1 :V3]) ;; [:V1 :V3] (tc/column-names DS #".*[14]") ;; regex: ends with 1 or 4 (tc/column-names DS :type/integer) ;; all integer columns (tc/column-names DS :type/float64) ;; float64 only (tc/column-names DS :!type/float64) ;; complement: everything but float64 (tc/column-names DS (complement #{:V1})) ;; all but :V1 ;; Match on both name and datatype simultaneously (tc/column-names DS (fn [meta] (and (= :int64 (:datatype meta)) (clojure.string/ends-with? (:name meta) "1"))) :all) ``` -------------------------------- ### Column Creation & Identity Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates how to create columns, check for column types, and create columns of ones or zeros. ```APIDOC ## Column API ### Creation & Identity Creating an empty column: ```clojure (tcc/column) ``` Creating a Column from a vector or a sequence: ```clojure (tcc/column [1 2 3 4 5]) (tcc/column `(1 2 3 4 5)) ``` Creating columns of ones or zeros: ```clojure (tcc/ones 10) (tcc/zeros 10) ``` Identifying a column using `column?`: ```clojure (tcc/column? [1 2 3 4 5]) (tcc/column? (tcc/column)) (tcc/column? (-> (tc/dataset {:a [1 2 3 4 5]}) :a)) ``` ``` -------------------------------- ### Get tech.ml.dataset Version Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the version of the tech.ml.dataset library from the deps.edn file. ```clojure (def tech-ml-version (get-in (read-string (slurp "deps.edn")) [:deps 'techascent/tech.ml.dataset :mvn/version])) ``` -------------------------------- ### Create a Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Creates a new dataset from a map of column names to sequences. Use this to initialize your data for manipulation. ```clojure (def DS (tc/dataset {:V1 (take 9 (cycle [1 2])) :V2 (range 1 10) :V3 (take 9 (cycle [0.5 1.0 1.5])) :V4 (take 9 (cycle ["A" "B" "C"]))})) ``` -------------------------------- ### Rename Columns Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Renames one or more columns in the dataset. This example renames :V2 to 'v2'. ```clojure (def DS (tc/rename-columns DS {:V2 "v2"})) ``` -------------------------------- ### Generate Detailed Documentation with Quarto Source: https://github.com/scicloj/tablecloth/blob/master/README.md Use this command to generate detailed documentation. Requires Quarto CLI v1.5.10 pre-release. ```clojure (require '[scicloj.clay.v2.api :as clay]) (clay/make! {:format [:quarto :html] :source-path "notebooks/index.clj"}) ``` -------------------------------- ### Get Dataset Name Source: https://context7.com/scicloj/tablecloth/llms.txt Retrieve the name assigned to the dataset. The name can be updated immutably. ```clojure (def ds (tc/dataset "https://vega.github.io/vega-lite/examples/data/seattle-weather.csv")) (tc/dataset-name ds) ;; => "seattle-weather" ;; Rename the dataset (immutable) (->> "seattle-weather" (tc/set-dataset-name ds) (tc/dataset-name)) ;; => "seattle-weather" ``` -------------------------------- ### Select columns by name pattern (starts-with) Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Selects columns whose names start with a specific prefix. This uses a predicate function to filter columns based on their names. ```clojure (tc/select-columns DS #(clojure.string/starts-with? (name %) "V")) ``` -------------------------------- ### Get Dataset Class Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the class of the dataset object. Useful for understanding the underlying implementation. ```clojure (class DS) ``` -------------------------------- ### Perform Left Join with Swapped Datasets Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Performs a left join between ds2 and ds1 on the ':b' column. This demonstrates joining in the reverse order compared to the previous example. ```clojure (tc/left-join ds2 ds1 :b) ``` -------------------------------- ### Get Dataset Name Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the current name associated with the dataset. This is often a URL or a descriptive string. ```clojure (tc/dataset-name ds) ``` -------------------------------- ### Create a Dataset for Crosstab Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Initializes a dataset used for demonstrating crosstab functionality. This dataset contains columns 'a', 'b', and 'c'. ```clojure (def ctds (tc/dataset {:a [:foo :foo :bar :bar :foo :foo] :b [:one :one :two :one :two :one] :c [:dull :dull :shiny :dull :dull :shiny]})) ``` -------------------------------- ### Get Dataset Row and Column Counts Source: https://context7.com/scicloj/tablecloth/llms.txt Retrieve the number of rows and columns in a dataset, or its shape as a vector. ```clojure (def ds (tc/dataset "https://vega.github.io/vega-lite/examples/data/seattle-weather.csv")) (tc/row-count ds) ;; => 1461 (tc/column-count ds) ;; => 6 (tc/shape ds) ;; => [1461 6] ``` -------------------------------- ### Select column names when ':all' is a column name Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html To select a column literally named `:all`, or a sequence/map, enclose it in a vector. This example returns an empty sequence as no such column exists. ```clojure (tc/column-names DS [:all]) ``` -------------------------------- ### Load Production Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Loads the production dataset for demonstrating pivoting with multiple column selectors. ```clojure (def production (tc/dataset "data/production.csv")) production ``` -------------------------------- ### Group and aggregate by multiple columns Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Groups the dataset by combinations of values from multiple columns and aggregates data within each group. This example calculates the sum of ':V2' for each unique pair of ':V4' and ':V1'. ```clojure (-> DS (tc/group-by [:V4 :V1]) (tc/aggregate {:sumV2 #(dfn/sum (% :V2))})) ``` -------------------------------- ### Load, Group, Aggregate, Order, and Head Dataset Source: https://github.com/scicloj/tablecloth/blob/master/README.md Demonstrates a common data processing pipeline: loading a CSV dataset, grouping by symbol and year, aggregating by mean price, ordering the results, and taking the first 10 rows. Requires the `tablecloth.api` and `tech.v3.datatype` libraries. ```clojure (-> "https://raw.githubusercontent.com/techascent/tech.ml.dataset/master/test/data/stocks.csv" (tc/dataset {:key-fn keyword}) (tc/group-by (fn [row] {:symbol (:symbol row) :year (tech.v3.datatype.datetime/long-temporal-field :years (:date row))})) (tc/aggregate #(tech.v3.datatype.functional/mean (% :price))) (tc/order-by [:symbol :year]) (tc/head 10)) ``` -------------------------------- ### Create Dataset from Map with Multiple Columns Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Create a dataset with multiple columns by providing a map where keys are column names and values are sequences. ```clojure (tc/dataset {:A [3 4 5] :B ["X" "Y" "Z"]}) ``` -------------------------------- ### Create Sample Dataset with Random Data Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Creates a sample dataset named 'pnl' with predefined columns and randomly generated data for 'y1' and 'y2'. This is useful for testing data manipulation functions. ```clojure (def pnl (tc/dataset {:x [1 2 3 4] :a [1 1 0 0] :b [0 1 1 1] :y1 (repeatedly 4 rand) :y2 (repeatedly 4 rand) :z1 [3 3 3 3] :z2 [-2 -2 -2 -2]})) pnl ``` -------------------------------- ### Pivot to Wider Format on Time Column Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Pivots a dataset to a wider format using the 'time' column as the new column headers. This example selects a subset of rows before pivoting. ```clojure (-> stocks-long (tc/select-rows (range 0 30 4)) (tc/pivot->wider "time" :price {:drop-missing? false})) ``` -------------------------------- ### Select column names using regex Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use a regular expression with `column-names` to select column names that match the pattern. This example selects columns ending with '1' or '4'. ```clojure (tc/column-names DS #".*[14]") ``` -------------------------------- ### Get First Row of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the first row of a dataset. This function is applicable to both regular and grouped datasets. ```clojure (tc/first DS) ``` -------------------------------- ### Create Dataset using let-dataset Macro Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html The `tc/let-dataset` macro allows defining columns similarly to R's `tibble`, enabling references to previously defined columns. ```clojure (tc/let-dataset [x (range 1 6) y 1 z (dfn/+ x y)]) ``` -------------------------------- ### Cast data (long to wide) with pivot->wider and string coercion Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html This example shows `pivot->wider` after coercing a value column to strings, then aggregating counts of string values. ```clojure (-> mDS (tc/map-columns :value #(str (> % 5))) ;; coerce to strings (tc/pivot->wider :value :variable {:fold-fn vec}) (tc/update-columns ["true" "false"] (partial map #(if (sequential? %) (count %) 1)))) ``` -------------------------------- ### Group and Aggregate by Key Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Groups rows by a key column and then applies an aggregation function to another column within each group. This example sums :V1 for each :V4 group, excluding 'B'. ```clojure (-> DS (tc/select-rows (comp (complement #{"B"}) :V4)) (tc/group-by [:V4]) (tc/aggregate-columns :V1 dfn/sum)) ``` -------------------------------- ### Create Empty Dataset with Named Columns Source: https://context7.com/scicloj/tablecloth/llms.txt Initialize an empty dataset with specified column names. ```clojure ;; Empty dataset with named columns (tc/dataset nil {:column-names [:a :b]}) ``` -------------------------------- ### Dataset Creation — tc/let-dataset Source: https://context7.com/scicloj/tablecloth/llms.txt Creates a dataset using `let`-style bindings where each symbol becomes a column; later bindings can refer to earlier ones. ```APIDOC ## tc/let-dataset ### Description Creates a dataset using `let`-style bindings where each symbol becomes a column; later bindings can refer to earlier ones. ### Method `tc/let-dataset` ### Parameters #### Path Parameters - `bindings` (vector) - A vector of bindings defining columns and their values. ### Request Example ```clojure (require '[tech.v3.datatype.functional :as dfn]) (tc/let-dataset [x (range 1 6) y 1 z (dfn/+ x y)]) ``` ``` -------------------------------- ### Load Fish Encounters Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Loads the fish encounters dataset for demonstration purposes. ```clojure (def fish (tc/dataset "data/fish_encounters.csv")) fish ``` -------------------------------- ### Get Column Count of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the total number of columns in a dataset. Useful for understanding the dimensionality of the data. ```clojure (tc/column-count ds) ``` -------------------------------- ### Create Empty Dataset with Column Names Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Provide column names using the `:column-names` option when creating an empty dataset. ```clojure (tc/dataset nil {:column-names [:a :b]}) ``` -------------------------------- ### Access Column Element by Index Source: https://github.com/scicloj/tablecloth/blob/master/docs/column_api.html Access elements in a column using `get` or `nth`, similar to Clojure vectors. ```clojure (-> (tcc/column [1 2 3 4 5]) (get 3)) ``` ```clojure (-> (tcc/column [1 2 3 4 5]) (nth 3)) ``` -------------------------------- ### Load WHO Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Loads the WHO dataset from a gzipped CSV file. This is a common starting point for analyzing health-related data. ```clojure (def who (tc/dataset "data/who.csv.gz")) ``` -------------------------------- ### Updating Columns with Transformations Source: https://context7.com/scicloj/tablecloth/llms.txt Demonstrates how to apply transformation functions to selected columns using `update-columns`. This includes reversing all columns, applying sequential transformations to numerical columns, and applying different functions to specific columns. ```clojure ;; Reverse all columns (tc/update-columns DS :all reverse) ;; Apply dec then inc to all numerical columns (two functions applied in sequence) (tc/update-columns DS :type/numerical [(partial map dec) (partial map inc)]) ;; Apply different functions per column via a map (tc/update-columns DS {:V1 reverse :V2 (comp shuffle seq)}) ``` -------------------------------- ### Reorder Columns Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Reorders the columns in the dataset to a specified sequence. This example places :V4 first, followed by :V1 and :V2. ```clojure (def DS (tc/reorder-columns DS :V4 :V1 :V2)) ``` -------------------------------- ### Create a dataset in Clojure Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Shows how to create a dataset in Clojure using the `tablecloth` library. The `:string` datatype is inferred for the ID column. ```Clojure (def DT (tc/dataset {:ID ["b" "b" "b" "a" "a" "c"] :a (range 1 7) :b (range 7 13) :c (range 13 19)})) DT ``` ```Clojure (-> :ID DT meta :datatype) ``` -------------------------------- ### Pivot to Wider Format with Custom Value Concatenation Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html This example demonstrates reshaping data to a wider format while customizing how values are concatenated. It uses `:concat-value-with vector` to combine values into a vector and `:drop-missing? false` to retain all rows. ```clojure (tc/pivot->wider income "variable" ["estimate" "moe"] {:concat-columns-with vec :concat-value-with vector :drop-missing? false}) ``` -------------------------------- ### Get Last N Rows of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the last N rows of a dataset. If N is not specified, it defaults to 5. ```clojure (tc/tail DS) ``` -------------------------------- ### Load Dataset with Tablecloth Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Loads a dataset from a CSV file using the Tablecloth library. This is a common first step before data manipulation. ```clojure (def family (tc/dataset "data/family.csv")) family ``` -------------------------------- ### Get First N Rows of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the first N rows of a dataset. If N is not specified, it defaults to 5. ```clojure (tc/head DS) ``` -------------------------------- ### List all available datatypes Source: https://github.com/scicloj/tablecloth/blob/master/docs/column_api.html Retrieves a list of all supported datatypes within the tech.ml.dataset system. These are the types that columns can be instantiated with. ```clojure (tech.v3.datatype.casting/all-datatypes) ``` -------------------------------- ### Get Random Row from Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Selects a single random row from the dataset. For reproducible results, a `:seed` option can be provided. ```clojure (tc/rand-nth DS) ``` ```clojure (tc/rand-nth DS {:seed 42}) ``` -------------------------------- ### Get Last Row of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the last row of a dataset. This function operates correctly on both regular and grouped datasets. ```clojure (tc/last DS) ``` -------------------------------- ### Get Keys of Grouped Dataset as Map Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Extracts the keys (group names) from a dataset grouped by :V1, returned as a map. ```clojure (keys (tc/group-by DS :V1 {:result-type :as-map})) ``` -------------------------------- ### Accessing Dataset Columns and Rows Source: https://context7.com/scicloj/tablecloth/llms.txt Demonstrates various ways to access columns and rows from a dataset, including single columns, columns as sequences or maps, and rows as sequences or maps. Also shows how to extract specific cell values and numerical columns as double arrays. ```clojure (def DS (tc/dataset {:V1 (take 9 (cycle [1 2])) :V2 (range 1 10) :V3 (take 9 (cycle [0.5 1.0 1.5])) :V4 (take 9 (cycle ["A" "B" "C"]}))) ;; Single column (returns a column object) (tc/column DS :V2) (DS :V2) ;; also works ;; Columns as sequence (take 2 (tc/columns DS)) ;; Columns as map of name → column (keys (tc/columns DS :as-map)) ;; Columns as vectors (tc/columns DS :as-vecs) ;; Rows as sequence of sequences (default) (take 2 (tc/rows DS)) ;; Rows as sequence of maps (take 2 (tc/rows DS :as-maps)) ;; Rows with missing values elided from maps (-> {:a [1 nil 2] :b [3 4 nil]} (tc/dataset) (tc/rows :as-maps {:nil-missing? false})) ;; Numerical columns as double-double-array (-> DS (tc/select-columns :type/numerical) (tc/head) (tc/rows :as-double-arrays)) ;; Single cell value (tc/get-entry DS :V2 3) ;; => 4 (get-in DS [:V2 3]) ;; => 4 ``` -------------------------------- ### Create Dataset from CSV URL Source: https://context7.com/scicloj/tablecloth/llms.txt Load a dataset directly from a CSV file located at a URL. Supports custom key-function for column names. ```clojure ;; From a CSV URL with keyword column names (tc/dataset "https://vega.github.io/vega-lite/examples/data/seattle-weather.csv" {:key-fn keyword}) ``` -------------------------------- ### Get Columns as Sequence Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieve all columns from a dataset as a sequence of column objects. Useful for iterating or taking a subset of columns. ```clojure (take 2 (tc/columns ds)) ``` -------------------------------- ### Get Row Count of Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the total number of rows in a dataset. This is a fundamental metric for understanding dataset size. ```clojure (tc/row-count ds) ``` -------------------------------- ### Define datasets for binding Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Defines three sample datasets (x, y, and z) with different column structures. These datasets are used to demonstrate various binding operations. ```clojure (def x (tc/dataset {:V1 [1 2 3]})) ``` ```clojure (def y (tc/dataset {:V1 [4 5 6]})) ``` ```clojure (def z (tc/dataset {:V1 [7 8 9] :V2 [0 0 0]})) ``` -------------------------------- ### Concatenate Grouped Datasets Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Concatenates grouped datasets, resulting in a grouped dataset. This example concatenates datasets grouped by different columns. ```clojure (tc/concat (tc/group-by DS [:V3]) (tc/group-by DS [:V4])) ``` -------------------------------- ### Create Dataset from Local File Source: https://context7.com/scicloj/tablecloth/llms.txt Read data into a dataset from a local CSV or other supported file formats. ```clojure ;; From a local CSV file (tc/dataset "data/iris.csv") ;; From an Excel file (requires dataset-io or fastexcel on classpath) (tc/dataset "data/singleSheet.xlsx") ;; From Parquet (-> (tc/dataset "data/titanic.parquet") (tc/select-columns ["PassengerId" "Pclass" "Age" "Survived"])) ``` -------------------------------- ### Load and Select Columns from CSV Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Loads a CSV dataset and selects the first 8 columns. This is a common initial step for exploring large datasets. ```clojure (def world-bank-pop (tc/dataset "data/world_bank_pop.csv.gz")) (->> world-bank-pop (tc/column-names) (take 8) (tc/select-columns world-bank-pop)) ``` -------------------------------- ### Create Dataset from Array of Int Arrays (As Columns) Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use the `:layout :as-columns` option to interpret inner arrays as columns when creating a dataset from an array of arrays. ```clojure (-> (map int-array [[1 2] [3 4] [5 6]]) (into-array) (tc/dataset {:layout :as-columns})) ``` -------------------------------- ### Get the first two rows of flights in Clojure Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Selects the first two rows from the `flights` dataset using `tc/select-rows` with a range. ```Clojure (tc/select-rows flights (range 2)) ``` -------------------------------- ### Define dataset x for joins Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Defines a sample dataset `x` used in join operations. This dataset contains 'Id', 'X1', and 'XY' columns. ```clojure (def x (tc/dataset {"Id" ["A" "B" "C" "C"] "X1" [1 3 5 7] "XY" ["x2" "x4" "x6" "x8"]})) ``` -------------------------------- ### Get Rows as Sequence of Maps Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieve rows from a dataset as a sequence of maps. Each map represents a row with column names as keys. ```clojure (clojure.pprint/pprint (take 2 (tc/rows ds :as-maps))) ``` -------------------------------- ### Import CSV File Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Load a dataset directly from a CSV file using its path. ```clojure (tc/dataset "data/family.csv") ``` -------------------------------- ### Slice Column Data Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Extract a contiguous sub-section of a column using `tcc/slice`. You can specify the start and end indices, and an optional step. ```clojure (-> (tcc/column (range 10)) (tcc/slice 5)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice 1 4)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice 0 9 2)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice :start :end 2)) ``` -------------------------------- ### Select columns while joining Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Demonstrates how to select specific columns from two datasets before performing a right join. This is useful for controlling the output schema and performance. ```clojure (tc/right-join (tc/select-columns x ["Id" "X1"]) (tc/select-columns y ["Id" "XY"]) "Id") ``` -------------------------------- ### Order by Custom Function and Column Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Orders the dataset 'DS' first by ':V4' in descending order, then by a custom function that calculates the product of ':V1', ':V2', and ':V3' in ascending order. ```clojure (tc/order-by DS [:V4 (fn [row] (* (:V1 row) (:V2 row) (:V3 row)))] [:desc :asc]) ``` -------------------------------- ### Column Arithmetic Operations Source: https://github.com/scicloj/tablecloth/blob/master/docs/column_api.html Demonstrates basic arithmetic operations like subtraction and exponentiation on columns. Ensure columns are compatible for the operation. ```clojure (def a (tcc/column [20 30 40 50])) (def b (tcc/column (range 4))) (tcc/- a b) ``` ```clojure (tcc/pow a 2) ``` ```clojure (tcc/* 10 (tcc/sin a)) ``` -------------------------------- ### Operate on grouped dataset as regular Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use the `without-grouping->` threading macro to operate on a grouped dataset as if it were a regular dataset, for example, to access columns. ```clojure (-> DS (tc/group-by [:V4 :V1]) (tc/without-grouping-> (tc/order-by (comp (juxt :V4 :V1) :name)))) ``` -------------------------------- ### Select all column names Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use `column-names` with the `:all` keyword or simply without arguments to select all column names from a dataset. ```clojure (tc/column-names DS) ``` ```clojure (tc/column-names DS :all) ``` -------------------------------- ### Get the first two rows of flights in R Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Selects the first two rows from the `flights` dataset. Requires the `knitr` package for `kable`. ```R ans <- flights[1:2] kable(ans) ``` -------------------------------- ### Split Dataset to Sequence (Bootstrap) Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Converts a dataset split into a sequence of pairs using a bootstrap strategy with specified repeats, and takes the first pair. ```clojure (-> for-splitting (tc/group-by :group) (tc/split->seq :bootstrap {:partition-selector :partition :seed 11 :ratio 0.8 :repeats 2}) (first)) ``` -------------------------------- ### Get Shape of Flights Data in Clojure Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the shape (number of rows and columns) of the flights dataset in Clojure using the `tc/shape` function. ```clojure (tc/shape flights) ``` -------------------------------- ### Get Dimensions of Flights Data in R Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieves the dimensions (number of rows and columns) of the flights dataset in R using the `dim` function. ```r dim(flights) ``` -------------------------------- ### Create Dataset from Array of Int Arrays (As Rows with Names) Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Specify column names using `:column-names` when creating a dataset from an array of arrays with the `:layout :as-rows` option. ```clojure (-> (map int-array [[1 2] [3 4] [5 6]]) (into-array) (tc/dataset {:layout :as-rows :column-names [:a :b]})) ``` -------------------------------- ### Create Composable Pipelines - `tablecloth.pipeline` Source: https://context7.com/scicloj/tablecloth/llms.txt Shows how to define functional pipelines using `tablecloth.pipeline` and `clojure.core/comp`. These pipelines can be applied to datasets. ```clojure (require '[tablecloth.pipeline :as tpipe]) ;; Functional pipeline: compose operations into a single fn (def pipeline (comp (tpipe/select-columns [:V1 :V2 :V3]) (tpipe/drop-missing) (tpipe/order-by :V2) (tpipe/head 5))) ;; Apply pipeline to any dataset (pipeline DS) ``` -------------------------------- ### Get Column Type Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use `tcc/typeof` to determine the data type of a column. The system attempts to auto-detect the type upon column creation. ```clojure (-> (tcc/column [1 2 3 4 5]) (tcc/typeof)) ``` ```clojure (-> (tcc/column [:a :b :c :d :e]) (tcc/typeof)) ``` ```clojure (-> (tcc/column [1 :b 3 :c 5]) (tcc/typeof)) ``` -------------------------------- ### Get Rows as Sequence of Sequences Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Extract rows from a dataset as a sequence of sequences. Each inner sequence represents a row's values in order. ```clojure (take 2 (tc/rows ds)) ``` -------------------------------- ### Slice Column Data Source: https://github.com/scicloj/tablecloth/blob/master/docs/column_api.html Extract a contiguous subset of a column using `tcc/slice`. Supports start, end, and step arguments, and keyword aliases. ```clojure (-> (tcc/column (range 10)) (tcc/slice 5)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice 1 4)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice 0 9 2)) ``` ```clojure (-> (tcc/column (range 10)) (tcc/slice :start :end 2)) ``` -------------------------------- ### Create Dataset from Map Source: https://context7.com/scicloj/tablecloth/llms.txt Construct a dataset from a map where keys are column names and values are sequences of data. Non-sequential values are broadcast. ```clojure ;; From a map of column-name → values (tc/dataset {:A [1 2 3] :B ["x" "y" "z"]}) ;; _unnamed [3 2]: :A :B ;; Non-sequential values are broadcast across all rows (tc/dataset {:A [1 2 3 4 5 6] :B "X" :C :a}) ``` -------------------------------- ### Split a column with separate-column Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Use `separate-column` to split a single column into multiple columns based on a delimiter. This example splits a string column by a colon. ```clojure (-> {:a ["A:a" "B:b" "C:c"]} (tc/dataset) (tc/separate-column :a [:V1 :V2 ":"]))) ``` -------------------------------- ### Revert Column Rename Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Reverts a column rename operation by renaming the column back to its original name. This example renames 'v2' back to :V2. ```clojure (def DS (tc/rename-columns DS {"v2" :V2})) ``` -------------------------------- ### Select Rows by Key Value Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Selects rows where the value in the specified key column matches a given condition. This example filters for rows where :V4 is 'A'. ```clojure (tc/select-rows DS #(= (:V4 %) "A")) ``` -------------------------------- ### Create Singleton Dataset Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Creates a singleton dataset when other import methods fail. This will typically result in an error message from the underlying tech.ml.dataset library. ```clojure (tc/dataset 999) ``` -------------------------------- ### Adding and Updating Columns Source: https://context7.com/scicloj/tablecloth/llms.txt Shows how to add new columns or replace existing ones using `add-column` and `add-columns`. The new column's value can be a scalar, sequence, another column, or a function. It also covers strategies for handling column size differences and applying functions per group. ```clojure ;; Add a scalar column (broadcast to all rows) (tc/add-column DS :V5 "X") ;; Add column by copying another (tc/add-column DS :V5 (DS :V1)) ;; Add column using a function on the dataset (tc/add-column DS :row-count tc/row-count) ;; Size strategies when new column is shorter than dataset (tc/add-column DS :V5 [:r :b] :cycle) ;; cycle [:r :b] to fill all rows (tc/add-column DS :V5 [:r :b] :na) ;; fill remainder with missing ;; Add multiple columns at once (tc/add-columns DS {:V1 #(map inc (% :V1)) :V5 #(map (comp keyword str) (% :V4)) :V6 11}) ;; Function on grouped dataset applies per group (-> DS (tc/group-by :V1) (tc/add-column :row-count tc/row-count) (tc/ungroup)) ``` -------------------------------- ### Get Single Value using get-in Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Retrieve a single value from the dataset using the `get-in` function, specifying the column name and row index. ```clojure (get-in ds ["wind" 2]) ``` -------------------------------- ### Perform SQL-style Joins on Datasets Source: https://context7.com/scicloj/tablecloth/llms.txt Use various join types like left, right, inner, full, semi, and anti joins to combine datasets based on specified columns. Supports joining on single or multiple columns, and with different column names on each side. ```clojure (def ds1 (tc/dataset {:a [1 2 1 2 3 4 nil nil 4] :b (range 101 110) :c (map str "abs tract")})) (def ds2 (tc/dataset {:a [nil 1 2 5 4 3 2 1 nil] :b (range 110 101 -1) :c (map str "datatable") :e [3 4 5 6 7 nil 8 1 1]})) ;; Left join on single column (tc/left-join ds1 ds2 :b) ;; Left join on multiple columns (tc/left-join ds1 ds2 [:a :b]) ;; Left join with different column names per side (tc/left-join ds1 ds2 {:left :a :right :e}) ;; Other join types (tc/right-join ds1 ds2 :b) (tc/inner-join ds1 ds2 :b) (tc/full-join ds1 ds2 :b) (tc/semi-join ds1 ds2 :b) ;; rows in ds1 that match ds2 (tc/anti-join ds1 ds2 :b) ;; rows in ds1 that don't match ds2 ``` -------------------------------- ### Create Dataset with `let-dataset` Source: https://context7.com/scicloj/tablecloth/llms.txt Define a dataset using `let`-style bindings, allowing later bindings to reference earlier ones. Requires `tech.v3.datatype.functional`. ```clojure (require '[tech.v3.datatype.functional :as dfn]) (tc/let-dataset [x (range 1 6) y 1 z (dfn/+ x y)]) ;; _unnamed [5 3]: ;; | :x | :y | :z | ;; |----|----|----| ;; | 1 | 1 | 2 | ;; | 2 | 1 | 3 | ;; | 3 | 1 | 4 | ;; | 4 | 1 | 5 | ;; | 5 | 1 | 6 | ``` -------------------------------- ### Get Columns as Map Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Access all columns from a dataset as a map where keys are column names and values are column data. Useful for direct access by name. ```clojure (keys (tc/columns ds :as-map)) ``` -------------------------------- ### Get Dataset Column Metadata Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Fetches metadata for each column in the dataset, including name, data type, element count, and whether the column is categorical. ```clojure (tc/info ds :columns) ``` -------------------------------- ### Join Columns - With Options Source: https://context7.com/scicloj/tablecloth/llms.txt Joins columns with custom separators and substitutes for missing values. Specify `{:drop-columns? false}` to retain original columns. ```clojure (tc/join-columns DS :joined [:V1 :V2 :V4] {:separator "/" :missing-subst "."}) ``` ```clojure ;; Keep source columns (tc/join-columns DS :joined [:V1 :V2 :V4] {:drop-columns? false}) ``` -------------------------------- ### Get Dataset Shape Source: https://github.com/scicloj/tablecloth/blob/master/docs/index.html Returns the shape of the dataset as a vector containing the row count and column count, in that order. [row count, column count]. ```clojure (tc/shape ds) ``` -------------------------------- ### Select Column Elements - `tablecloth.column.api` Source: https://context7.com/scicloj/tablecloth/llms.txt Demonstrates selecting elements from a column based on indices or a boolean mask. Also shows selecting based on a predicate like `tcc/odd?`. ```clojure (require '[tablecloth.column.api :as tcc]) ;; Select (discontinuous subset) (tcc/select (tcc/column (range 10)) [1 9]) ;; by index (tcc/select (tcc/column (range 10)) (tcc/column [true false true])) ;; by boolean mask ;; Boolean-based selection (let [col (tcc/column (range 10))] (->> col (tcc/odd?) (tcc/select col))) ;; select only odd values ```