### Start Marquez with Sample Data (MacOS/Linux) Source: https://marquezproject.ai/docs/quickstart Starts the Marquez HTTP server using Docker with sample data for demonstration purposes on MacOS and Linux. The `--seed` flag populates Marquez with initial metadata. ```bash ./docker/up.sh --seed ``` -------------------------------- ### Start Marquez with Sample Data (Windows) Source: https://marquezproject.ai/docs/quickstart Starts the Marquez HTTP server using Docker with sample data for demonstration purposes on Windows. The `--seed` flag populates Marquez with initial metadata. Assumes Postgres and Bash are in the PATH. ```bash sh ./docker/up.sh --seed ``` -------------------------------- ### Retrieve Dataset Java OkHttp Example Source: https://marquezproject.ai/docs/api/get-dataset This example demonstrates retrieving a dataset using Java with the OkHttp library. It constructs an HTTP client and sends a GET request to the API. ```java import okhttp3.*; import java.io.IOException; public class DatasetClient { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); String url = "http://localhost:5000/api/v1/namespaces/your_namespace/datasets/your_dataset"; Request request = new Request.Builder() .url(url) .header("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { if (response.isSuccessful()) { System.out.println(response.body().string()); } else { System.out.println("Error: " + response.code()); } } } } ``` -------------------------------- ### List Sources API Request Examples Source: https://marquezproject.ai/docs/api/get-sources Examples of how to make a GET request to the /sources endpoint using various programming languages and tools. These examples demonstrate how to fetch a list of sources from the Marquez Project AI API. ```curl curl -L -X GET 'http://localhost:5000/api/v1/sources' \ -H 'Accept: application/json' ``` ```python import requests url = "http://localhost:5000/api/v1/sources" headers = { 'Accept': 'application/json' } response = requests.get(url, headers=headers) print(response.json()) ``` ```go package main import ( "fmt" "net/http" "io/ioutil" ) func main() { client := &http.Client{} req, err := http.NewRequest("GET", "http://localhost:5000/api/v1/sources", nil) if err != nil { fmt.Println(err) return } req.Header.Add("Accept", "application/json") res, err := client.Do(req) if err != nil { fmt.Println(err) return } defer res.Body.Close() body, err := ioutil.ReadAll(res.Body) if err != nil { fmt.Println(err) return } fmt.Println(string(body)) } ``` ```nodejs const axios = require('axios'); const url = 'http://localhost:5000/api/v1/sources'; const headers = { 'Accept': 'application/json' }; axios.get(url, { headers: headers }) .then(response => { console.log(response.data); }) .catch(error => { console.error(error); }); ``` ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/sources') http = Net::HTTP.new(uri.host, uri.port) request = Net::HTTP::Get.new(uri.request_uri) request['Accept'] = 'application/json' response = http.request(request) puts response.body ``` ```csharp using RestSharp; using System; public class Example { public static void Main(string[] args) { var client = new RestClient("http://localhost:5000/api/v1"); var request = new RestRequest("/sources", Method.GET); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); Console.WriteLine(response.Content); } } ``` ```php ``` ```java import okhttp3.OkHttpClient; import okhttp3.Request; import okhttp3.Response; import java.io.IOException; public class Example { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/sources") .header("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { if (!response.isSuccessful()) throw new IOException("Unexpected code " + response); System.out.println(response.body().string()); } } } ``` ```powershell Invoke-RestMethod -Uri 'http://localhost:5000/api/v1/sources' -Headers @{Accept='application/json'} -Method Get ``` -------------------------------- ### Query Datasets and Jobs API Request Examples Source: https://marquezproject.ai/docs/api/search Provides example API requests for querying datasets and jobs. It demonstrates how to construct GET requests to the /search endpoint with various query parameters. The examples cover cURL, Python, Go, Node.js, Ruby, C#, PHP, Java, and PowerShell. ```curl curl -L -X GET 'http://localhost:5000/api/v1/search' \ -H 'Accept: application/json' ``` ```python import requests url = "http://localhost:5000/api/v1/search" headers = { 'Accept': 'application/json' } response = requests.get(url, headers=headers) print(response.json()) ``` ```go package main import ( "fmt" "net/http" ) func main() { client := &http.Client{} req, err := http.NewRequest("GET", "http://localhost:5000/api/v1/search", nil) if err != nil { fmt.Println(err) return } req.Header.Add("Accept", "application/json") resp, err := client.Do(req) if err != nil { fmt.Println(err) return } defer resp.Body.Close() // Process response body here fmt.Println("Status:", resp.Status) } ``` ```nodejs const axios = require('axios'); const url = 'http://localhost:5000/api/v1/search'; const headers = { 'Accept': 'application/json' }; axios.get(url, { headers: headers }) .then(response => { console.log(response.data); }) .catch(error => { console.error(error); }); ``` ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/search') request = Net::HTTP::Get.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port) do |http| http.request(request) end puts response.body ``` ```csharp using RestSharp; using System; public class Example { public static void Main(string[] args) { var client = new RestClient("http://localhost:5000/api/v1"); var request = new RestRequest("search", Method.GET); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); Console.WriteLine(response.Content); } } ``` ```php ``` ```java import okhttp3.*; import java.io.IOException; public class OkHttpExample { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/search") .addHeader("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { if (!response.isSuccessful()) throw new IOException("Unexpected code " + response); System.out.println(response.body().string()); } } } ``` ```powershell Invoke-RestMethod -Uri 'http://localhost:5000/api/v1/search' -Method Get -Headers @{"Accept"="application/json"} ``` -------------------------------- ### Retrieve Dataset Python Requests Example Source: https://marquezproject.ai/docs/api/get-dataset This example demonstrates how to retrieve a dataset using the Python 'requests' library. It constructs the URL and sends a GET request with the appropriate headers. ```python import requests base_url = "http://localhost:5000/api/v1" namespace = "your_namespace" dataset_name = "your_dataset" url = f"{base_url}/namespaces/{namespace}/datasets/{dataset_name}" headers = { "Accept": "application/json" } response = requests.get(url, headers=headers) if response.status_code == 200: print(response.json()) else: print(f"Error: {response.status_code}") ``` -------------------------------- ### Retrieve Dataset Ruby Net::HTTP Example Source: https://marquezproject.ai/docs/api/get-dataset This example shows how to retrieve a dataset using Ruby's built-in Net::HTTP library. It establishes a connection, sends a GET request, and processes the response. ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/namespaces/your_namespace/datasets/your_dataset') http = Net::HTTP.new(uri.host, uri.port) request = Net::HTTP::Get.new(uri.request_uri) request['Accept'] = 'application/json' response = http.request(request) if response.code == '200' puts response.body else puts "Error: #{response.code}" end ``` -------------------------------- ### List Tags API Request Examples Source: https://marquezproject.ai/docs/api/get-tags Examples of how to make an API request to list tags using different programming languages and tools. These examples demonstrate the GET request to the /tags endpoint with the specified base URL and Accept header. ```curl curl -L -X GET 'http://localhost:5000/api/v1/tags' \ -H 'Accept: application/json' ``` ```python import requests url = "http://localhost:5000/api/v1/tags" headers = { 'Accept': 'application/json' } response = requests.get(url, headers=headers) print(response.json()) ``` ```go package main import ( "fmt" "net/http" "io/ioutil" ) func main() { client := &http.Client{} req, err := http.NewRequest("GET", "http://localhost:5000/api/v1/tags", nil) if err != nil { fmt.Println(err) return } req.Header.Add("Accept", "application/json") resp, err := client.Do(req) if err != nil { fmt.Println(err) return } defer resp.Body.Close() body, err := ioutil.ReadAll(resp.Body) if err != nil { fmt.Println(err) return } fmt.Println(string(body)) } ``` ```nodejs const axios = require('axios'); const url = 'http://localhost:5000/api/v1/tags'; const headers = { 'Accept': 'application/json' }; axios.get(url, { headers }) .then(response => { console.log(response.data); }) .catch(error => { console.error(error); }); ``` ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/tags') request = Net::HTTP::Get.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => uri.scheme == 'https') do |http| http.request(request) end puts response.body ``` ```csharp using RestSharp; using System; public class Example { public static void Main(string[] args) { var client = new RestClient("http://localhost:5000/api/v1"); var request = new RestRequest("tags", Method.GET); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); Console.WriteLine(response.Content); } } ``` ```php ``` ```java import okhttp3.OkHttpClient; import okhttp3.Request; import okhttp3.Response; import java.io.IOException; public class Main { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/tags") .header("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { if (!response.isSuccessful()) throw new IOException("Unexpected code " + response); System.out.println(response.body().string()); } } } ``` ```powershell Invoke-RestMethod -Uri "http://localhost:5000/api/v1/tags" -Method Get -Headers @{"Accept"="application/json"} ``` -------------------------------- ### Retrieve Source API Request Examples Source: https://marquezproject.ai/docs/api/get-source Examples of how to retrieve a source using the Marquez Project AI API. This involves sending a GET request to the /sources/:source endpoint. The response includes details about the source such as its type, name, creation and update timestamps, connection URL, and description. ```curl curl -L -X GET 'http://localhost:5000/api/v1/sources/:source' \ -H 'Accept: application/json' ``` ```python import requests url = "http://localhost:5000/api/v1/sources/:source" headers = { 'Accept': 'application/json' } response = requests.get(url, headers=headers) print(response.json()) ``` ```go package main import ( "fmt" "net/http" "io/ioutil" ) func main() { client := &http.Client{} req, err := http.NewRequest("GET", "http://localhost:5000/api/v1/sources/:source", nil) if err != nil { fmt.Println(err) return } req.Header.Add("Accept", "application/json") res, err := client.Do(req) if err != nil { fmt.Println(err) return } defer res.Body.Close() body, err := ioutil.ReadAll(res.Body) if err != nil { fmt.Println(err) return } fmt.Println(string(body)) } ``` ```javascript const axios = require('axios'); const url = 'http://localhost:5000/api/v1/sources/:source'; const headers = { 'Accept': 'application/json' }; axios.get(url, { headers: headers }) .then(response => { console.log(response.data); }) .catch(error => { console.error(error); }); ``` ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/sources/:source') request = Net::HTTP::Get.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port) do |http| http.request(request) end puts response.body ``` ```csharp using RestSharp; using System; public class Example { public static void Main(string[] args) { var client = new RestClient("http://localhost:5000/api/v1"); var request = new RestRequest("sources/:source", Method.GET); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); Console.WriteLine(response.Content); } } ``` ```php ``` ```java import okhttp3.OkHttpClient; import okhttp3.Request; import okhttp3.Response; import java.io.IOException; public class Example { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/sources/:source") .get() .addHeader("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { System.out.println(response.body().string()); } } } ``` ```powershell Invoke-RestMethod -Uri "http://localhost:5000/api/v1/sources/:source" -Method Get -Headers @{"Accept"="application/json"} ``` -------------------------------- ### Retrieve Dataset Node.js Axios Example Source: https://marquezproject.ai/docs/api/get-dataset This example demonstrates retrieving a dataset using Node.js with the 'axios' library. It makes a GET request to the API endpoint and logs the response data or any errors. ```javascript const axios = require('axios'); const baseUrl = 'http://localhost:5000/api/v1'; const namespace = 'your_namespace'; const datasetName = 'your_dataset'; const url = `${baseUrl}/namespaces/${namespace}/datasets/${datasetName}`; axios.get(url, { headers: { 'Accept': 'application/json' } }) .then(response => { console.log(response.data); }) .catch(error => { console.error('Error fetching dataset:', error); }); ``` -------------------------------- ### Clone Marquez Repository (Windows) Source: https://marquezproject.ai/docs/quickstart Clones the Marquez source code repository from GitHub for Windows users, including a Git configuration to handle line endings. This is a prerequisite for running Marquez locally. ```bash git config --global core.autocrlf false git clone https://github.com/MarquezProject/marquez && cd marquez ``` -------------------------------- ### Start Run API Request (Go) Source: https://marquezproject.ai/docs/api/start-run This snippet illustrates how to start a run using the Marquez Project AI API with Go's native HTTP client. It constructs a POST request to the run start endpoint. ```go package main import ( "fmt" "net/http" ) func main() { url := "http://localhost:5000/api/v1/jobs/runs/:id/start" req, err := http.NewRequest("POST", url, nil) if err != nil { fmt.Printf("Error creating request: %s\n", err) return } req.Header.Set("Accept", "application/json") client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Printf("Error sending request: %s\n", err) return } defer resp.Body.Close() fmt.Printf("Status Code: %d\n", resp.StatusCode) } ``` -------------------------------- ### Start Run API Request (Java) Source: https://marquezproject.ai/docs/api/start-run This snippet demonstrates how to start a run using the Marquez Project AI API with Java and OkHttp. It constructs a POST request to the specified endpoint. ```java import okhttp3.*; import java.io.IOException; public class StartRun { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/jobs/runs/:id/start") .post(RequestBody.create(null, new byte[0])) // POST request with empty body .addHeader("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { System.out.println("Status Code: " + response.code()); } } } ``` -------------------------------- ### List Job Runs (Go Example) Source: https://marquezproject.ai/docs/api/get-runs This Go code snippet shows how to retrieve a list of runs for a job using the native HTTP client. It constructs the URL and sends a GET request, handling the JSON response. ```go package main import ( "fmt" "net/http" "io/ioutil" ) func main() { baseURL := "http://localhost:5000/api/v1" namespace := "your_namespace" job := "your_job" url := fmt.Sprintf("%s/namespaces/%s/jobs/%s/runs", baseURL, namespace, job) req, err := http.NewRequest("GET", url, nil) if err != nil { fmt.Printf("Error creating request: %s\n", err) return } req.Header.Set("Accept", "application/json") client := &http.Client{} resp, err := client.Do(req) if err != nil { fmt.Printf("Error sending request: %s\n", err) return } defer resp.Body.Close() if resp.StatusCode == http.StatusOK { body, err := ioutil.ReadAll(resp.Body) if err != nil { fmt.Printf("Error reading response body: %s\n", err) return } fmt.Println(string(body)) } else { fmt.Printf("Error: %d\n", resp.StatusCode) } } ``` -------------------------------- ### Start Run API Request (Ruby) Source: https://marquezproject.ai/docs/api/start-run This snippet shows how to start a run using the Marquez Project AI API with Ruby's Net::HTTP library. It constructs a POST request to the run start endpoint. ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/jobs/runs/:id/start') request = Net::HTTP::Post.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port) do |http| http.request(request) end puts "Status Code: #{response.code}" ``` -------------------------------- ### Clone Marquez Repository (MacOS/Linux) Source: https://marquezproject.ai/docs/quickstart Clones the Marquez source code repository from GitHub for MacOS and Linux users. This is a prerequisite for running Marquez locally. ```bash git clone https://github.com/MarquezProject/marquez && cd marquez ``` -------------------------------- ### Retrieve Namespace using Ruby Net::HTTP Source: https://marquezproject.ai/docs/api/get-namespace Shows how to retrieve a namespace using Ruby's built-in 'Net::HTTP' library. This example demonstrates making a GET request and parsing the JSON response. ```ruby require 'net/http' require 'uri' require 'json' uri = URI.parse('http://localhost:5000/api/v1/namespaces/:namespace') response = Net::HTTP.start(uri.host, uri.port) do |http| request = Net::HTTP::Get.new uri.request_uri request['Accept'] = 'application/json' http.request request end if response.code == '200' puts JSON.parse(response.body) else puts "Error: #{response.code}" end ``` -------------------------------- ### List Job Runs (Ruby Example) Source: https://marquezproject.ai/docs/api/get-runs This Ruby code snippet demonstrates how to fetch job runs using the Net::HTTP library. It constructs the API endpoint and sends a GET request, processing the JSON response. ```ruby require 'net/http' require 'uri' require 'json' base_url = 'http://localhost:5000/api/v1' namespace = 'your_namespace' job = 'your_job' uri = URI.parse("#{base_url}/namespaces/#{namespace}/jobs/#{job}/runs") response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http| request = Net::HTTP::Get.new(uri) request['Accept'] = 'application/json' http.request(request) end if response.code == '200' runs = JSON.parse(response.body) puts runs else puts "Error: #{response.code}" end ``` -------------------------------- ### Start Run API Request (C#) Source: https://marquezproject.ai/docs/api/start-run This snippet demonstrates how to start a run using the Marquez Project AI API with C# and the RestSharp library. It sends a POST request to the specified endpoint. ```csharp using RestSharp; var client = new RestClient("http://localhost:5000/api/v1/jobs/runs/:id/start"); var request = new RestRequest(Method.POST); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); Console.WriteLine(response.StatusCode); ``` -------------------------------- ### Retrieve Namespace using Go Native HTTP Source: https://marquezproject.ai/docs/api/get-namespace Provides an example of retrieving a namespace using Go's native 'net/http' package. This code makes a GET request and processes the JSON response. ```go package main import ( "fmt" "net/http" "io/ioutil" ) func main() { url := "http://localhost:5000/api/v1/namespaces/:namespace" req, err := http.NewRequest("GET", url, nil) if err != nil { fmt.Printf("Error creating request: %s\n", err) return } req.Header.Set("Accept", "application/json") client := &http.Client{} res, err := client.Do(req) if err != nil { fmt.Printf("Error sending request: %s\n", err) return } defer res.Body.Close() if res.StatusCode == http.StatusOK { body, err := ioutil.ReadAll(res.Body) if err != nil { fmt.Printf("Error reading response body: %s\n", err) return } fmt.Println(string(body)) } else { fmt.Printf("Error: %d\n", res.StatusCode) } } ``` -------------------------------- ### Start Run API Request (PHP) Source: https://marquezproject.ai/docs/api/start-run This snippet shows how to start a run using the Marquez Project AI API with PHP and cURL. It sends a POST request to the run start endpoint. ```php ``` -------------------------------- ### Retrieve a Run by ID (Go Example) Source: https://marquezproject.ai/docs/api/get-run This Go code snippet shows how to retrieve a run by its ID using the native 'net/http' package. It makes a GET request and handles the JSON response. ```go package main import ( "fmt" "io/ioutil" "net/http" ) func main() { baseURL := "http://localhost:5000/api/v1" runID := "YOUR_RUN_ID" // Replace with the actual run ID url := fmt.Sprintf("%s/jobs/runs/%s", baseURL, runID) req, err := http.NewRequest("GET", url, nil) if err != nil { fmt.Printf("Error creating request: %v\n", err) return } req.Header.Set("Accept", "application/json") client := &http.Client{} res, err := client.Do(req) if err != nil { fmt.Printf("Error sending request: %v\n", err) return } defer res.Body.Close() if res.StatusCode == http.StatusOK { body, err := ioutil.ReadAll(res.Body) if err != nil { fmt.Printf("Error reading response body: %v\n", err) return } fmt.Println(string(body)) } else { fmt.Printf("Error: %d - %s\n", res.StatusCode, res.Status) } } ``` -------------------------------- ### Start Run API Request (Node.js) Source: https://marquezproject.ai/docs/api/start-run This snippet demonstrates how to start a run using the Marquez Project AI API with Node.js and the 'axios' library. It sends a POST request to the specified endpoint. ```javascript const axios = require('axios'); const url = 'http://localhost:5000/api/v1/jobs/runs/:id/start'; const headers = { 'Accept': 'application/json' }; axios.post(url, null, { headers: headers }) .then(response => { console.log('Status Code:', response.status); }) .catch(error => { console.error('Error:', error); }); ``` -------------------------------- ### Start Run API Request (PowerShell) Source: https://marquezproject.ai/docs/api/start-run This snippet shows how to start a run using the Marquez Project AI API with PowerShell's Invoke-RestMethod. It sends a POST request to the run start endpoint. ```powershell $url = 'http://localhost:5000/api/v1/jobs/runs/:id/start' $headers = @{ 'Accept' = 'application/json' } $response = Invoke-RestMethod -Uri $url -Method Post -Headers $headers $response | ConvertTo-Json ``` -------------------------------- ### List Job Runs (Java Example) Source: https://marquezproject.ai/docs/api/get-runs This Java code snippet uses the OkHttp library to make a GET request to the Marquez API for listing job runs. It constructs the request with the correct URL and headers and processes the response. ```java import okhttp3.*; import java.io.IOException; public class MarquezClient { public static void listRuns(String baseUrl, String namespace, String job) { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url(String.format("%s/namespaces/%s/jobs/%s/runs", baseUrl, namespace, job)) .header("Accept", "application/json") .build(); try (Response response = client.newCall(request).execute()) { if (response.isSuccessful()) { System.out.println(response.body().string()); } else { System.out.println("Error: " + response.code()); } } catch (IOException e) { e.printStackTrace(); } } public static void main(String[] args) { String baseUrl = "http://localhost:5000/api/v1"; String namespace = "your_namespace"; String job = "your_job"; listRuns(baseUrl, namespace, job); } } ``` -------------------------------- ### Start Run API Request (Python) Source: https://marquezproject.ai/docs/api/start-run This snippet shows how to start a run using the Marquez Project AI API with the Python 'requests' library. It sends a POST request to the specified endpoint, including the run ID. ```python import requests url = "http://localhost:5000/api/v1/jobs/runs/:id/start" headers = { 'Accept': 'application/json' } response = requests.post(url, headers=headers) print(response.status_code) ``` -------------------------------- ### Get Column Lineage Graph - API Request Examples Source: https://marquezproject.ai/docs/api/get-column-lineage Examples of how to make a GET request to the /column-lineage endpoint using different programming languages and tools. These examples demonstrate how to specify query parameters like nodeId, depth, and withDownstream. ```curl curl -L -X GET 'http://localhost:5000/api/v1/column-lineage' \ -H 'Accept: application/json' ``` ```python import requests url = "http://localhost:5000/api/v1/column-lineage" headers = { 'Accept': 'application/json' } params = { 'nodeId': 'dataset:my-namespace:my-dataset', 'depth': 20, 'withDownstream': True } response = requests.get(url, headers=headers, params=params) print(response.json()) ``` ```go package main import ( "fmt" "net/http" ) func main() { url := "http://localhost:5000/api/v1/column-lineage?nodeId=dataset:my-namespace:my-dataset&depth=20&withDownstream=true" req, _ := http.NewRequest("GET", url, nil) req.Header.Set("Accept", "application/json") client := &http.Client{} res, err := client.Do(req) if err != nil { fmt.Println(err) return } defer res.Body.Close() // Process response body here } ``` ```nodejs const axios = require('axios'); const url = 'http://localhost:5000/api/v1/column-lineage'; const headers = { 'Accept': 'application/json' }; const params = { nodeId: 'dataset:my-namespace:my-dataset', depth: 20, withDownstream: true }; axios.get(url, { headers: headers, params: params }) .then(response => { console.log(response.data); }) .catch(error => { console.error(error); }); ``` ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/column-lineage?nodeId=dataset:my-namespace:my-dataset&depth=20&withDownstream=true') request = Net::HTTP::Get.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port) do |http| http.request(request) end puts response.body ``` ```csharp using RestSharp; using System; var client = new RestClient("http://localhost:5000/api/v1"); var request = new RestRequest("column-lineage", Method.GET); request.AddHeader("Accept", "application/json"); request.AddQueryParameter("nodeId", "dataset:my-namespace:my-dataset", ParameterType.Query); request.AddQueryParameter("depth", "20", ParameterType.Query); request.AddQueryParameter("withDownstream", "true", ParameterType.Query); var response = client.Execute(request); Console.WriteLine(response.Content); ``` ```php "http://localhost:5000/api/v1/column-lineage?nodeId=dataset:my-namespace:my-dataset&depth=20&withDownstream=true", CURLOPT_RETURNTRANSFER => true, CURLOPT_ENCODING => "", CURLOPT_MAXREDIRS => 10, CURLOPT_TIMEOUT => 30, CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1, CURLOPT_CUSTOMREQUEST => "GET", CURLOPT_HTTPHEADER => [ "Accept: application/json" ], ]); $response = curl_exec($curl); $err = curl_error($curl); curl_close($curl); if ($err) { echo "cURL Error #" . $err; } else { echo $response; } ``` ```java OkHttpClient client = new OkHttpClient().newBuilder() .build(); MediaType mediaType = MediaType.parse("application/json"); RequestBody body = RequestBody.create(mediaType, ""); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/column-lineage?nodeId=dataset:my-namespace:my-dataset&depth=20&withDownstream=true") .method("GET", null) .addHeader("Accept", "application/json") .build(); Response response = client.newCall(request).execute(); ``` ```powershell $url = "http://localhost:5000/api/v1/column-lineage?nodeId=dataset:my-namespace:my-dataset&depth=20&withDownstream=true" $headers = @{ "Accept" = "application/json" } Invoke-RestMethod -Method Get -Uri $url -Headers $headers ``` -------------------------------- ### Retrieve Dataset Go Native Example Source: https://marquezproject.ai/docs/api/get-dataset This example shows how to retrieve a dataset using Go's native HTTP client. It constructs the request, sets headers, and handles the response. ```go package main import ( "fmt" "io/ioutil" "net/http" ) func main() { url := "http://localhost:5000/api/v1/namespaces/your_namespace/datasets/your_dataset" req, err := http.NewRequest("GET", url, nil) if err != nil { fmt.Println("Error creating request:", err) return } req.Header.Set("Accept", "application/json") client := &http.Client{} res, err := client.Do(req) if err != nil { fmt.Println("Error sending request:", err) return } defer res.Body.Close() body, err := ioutil.ReadAll(res.Body) if err != nil { fmt.Println("Error reading response body:", err) return } if res.StatusCode == 200 { fmt.Println(string(body)) } else { fmt.Printf("Error: %d\n", res.StatusCode) } } ``` -------------------------------- ### Tag Dataset - Java OkHttp Example Source: https://marquezproject.ai/docs/api/add-tag-to-dataset Example of how to tag a dataset using Java with the OkHttp library. This code demonstrates building a request and executing it asynchronously. ```java import okhttp3.*; import java.io.IOException; public class TagDataset { public static void main(String[] args) throws IOException { OkHttpClient client = new OkHttpClient(); Request request = new Request.Builder() .url("http://localhost:5000/api/v1/namespaces/:namespace/datasets/:dataset/tags/:tag") .header("Accept", "application/json") .post(RequestBody.create(null, new byte[0])) // POST with empty body .build(); try (Response response = client.newCall(request).execute()) { System.out.println(response.code()); } } } ``` -------------------------------- ### Tag Dataset - Go Native Example Source: https://marquezproject.ai/docs/api/add-tag-to-dataset Example of how to tag a dataset using Go's native HTTP client. This code shows how to create a client, build the request, and send it to the API. ```go package main import ( "fmt" "net/http" ) func main() { url := "http://localhost:5000/api/v1/namespaces/:namespace/datasets/:dataset/tags/:tag" client := &http.Client{} req, err := http.NewRequest("POST", url, nil) if err != nil { fmt.Println(err) return } req.Header.Set("Accept", "application/json") resp, err := client.Do(req) if err != nil { fmt.Println(err) return } defer resp.Body.Close() fmt.Println(resp.StatusCode) } ``` -------------------------------- ### Retrieve Dataset PowerShell RestMethod Example Source: https://marquezproject.ai/docs/api/get-dataset This example shows how to retrieve a dataset using PowerShell's Invoke-RestMethod cmdlet. It sends a GET request to the specified API endpoint. ```powershell $url = "http://localhost:5000/api/v1/namespaces/your_namespace/datasets/your_dataset" $headers = @{ "Accept" = "application/json" } try { $response = Invoke-RestMethod -Uri $url -Method Get -Headers $headers Write-Host $response } catch { Write-Error "Error: $($_.Exception.Message)" } ``` -------------------------------- ### Start Marquez Services with Custom DB Port (Bash) Source: https://marquezproject.ai/docs/tutorials/airflow_tutorial Starts the Marquez services using Docker Compose, allowing for a custom port configuration for the database. This is useful to avoid port conflicts. ```bash ./docker/up.sh --db-port 2345 ``` ```bash sh ./docker/up.sh --db-port 2345 ``` -------------------------------- ### Run Object Schema Example (JSON) Source: https://marquezproject.ai/docs/api/create-run This JSON object represents the schema for a run within the Marquez Project AI. It includes fields for the run's ID, creation and update timestamps, nominal start and end times, current state, actual start and end times, duration, arguments, context, and facets. This example shows a completed run with specific arguments and context. ```JSON { "id": "870492da-ecfb-4be0-91b9-9a89ddd3db90", "createdAt": "2019-05-09T19:49:24.201361Z", "updatedAt": "2019-05-09T19:49:24.201361Z", "nominalStartTime": null, "nominalEndTime": null, "state": "COMPLETED", "startedAt": "2019-05-09T15:17:32.690346", "endedAt": "2019-05-09T20:05:46.815920Z", "durationMs": 4250894125, "args": { "email": "me@example.com", "emailOnFailure": "false", "emailOnRetry": "true", "retries": "1" }, "context": { "SQL": "SELECT * FROM mytable;" }, "facets": {} } ``` -------------------------------- ### Retrieve Namespace API Request Examples Source: https://marquezproject.ai/docs/api/get-namespace Examples of how to retrieve a namespace using the Marquez Project AI API. This involves sending a GET request to the /namespaces/:namespace endpoint. The response is a JSON object containing details about the namespace. ```json { "name": "my-namespace", "createdAt": "2019-05-09T19:49:24.201361Z", "updatedAt": "2019-05-09T19:49:24.201361Z", "ownerName": "me", "description": "My first namespace!" } ``` ```http GET /namespaces/:namespace ``` ```http http://localhost:5000/api/v1 ``` -------------------------------- ### Install Airflow OpenLineage Provider Package (Bash) Source: https://marquezproject.ai/docs/tutorials/airflow_tutorial Installs the necessary Apache Airflow OpenLineage Provider package to enable OpenLineage event emission. This command is used on MacOS/Linux systems. ```bash $ pip install apache-airflow-providers-openlineage ``` -------------------------------- ### Retrieve a Run by ID (PowerShell Example) Source: https://marquezproject.ai/docs/api/get-run This PowerShell script uses 'Invoke-RestMethod' to retrieve run details. It specifies the URI and headers for the GET request. ```powershell $baseURL = 'http://localhost:5000/api/v1' $runId = 'YOUR_RUN_ID' # Replace with the actual run ID $uri = "$baseURL/jobs/runs/$runId" $headers = @{ 'Accept' = 'application/json' } $response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers if ($response) { $response | ConvertTo-Json } else { Write-Error "Failed to retrieve run data." } ``` -------------------------------- ### Retrieve Dataset cURL Request Source: https://marquezproject.ai/docs/api/get-dataset Example of how to retrieve a dataset using cURL. This command sends a GET request to the specified endpoint with the 'Accept: application/json' header. ```curl curl -L -X GET 'http://localhost:5000/api/v1/namespaces/:namespace/datasets/:dataset' \ -H 'Accept: application/json' ``` -------------------------------- ### Clone Marquez Repository (Bash) Source: https://marquezproject.ai/docs/tutorials/airflow_tutorial Clones the Marquez source code repository from GitHub. This is a prerequisite for setting up and running Marquez. ```bash git clone https://github.com/MarquezProject/marquez && cd marquez ``` ```bash git config --global core.autocrlf false git clone https://github.com/MarquezProject/marquez && cd marquez ``` -------------------------------- ### Retrieve a Run by ID (cURL Example) Source: https://marquezproject.ai/docs/api/get-run This cURL command demonstrates how to retrieve a run by its ID. It sends a GET request to the specified endpoint with an 'Accept' header for JSON. ```shell curl -L -X GET 'http://localhost:5000/api/v1/jobs/runs/:id' \ -H 'Accept: application/json' ``` -------------------------------- ### Retrieve Dataset Example (JSON) Source: https://marquezproject.ai/docs/api/get-dataset This is an example JSON response when successfully retrieving a dataset. It includes metadata such as ID, type, name, creation timestamp, source information, fields, tags, and description. ```json { "id": { "namespace": "my-namespace", "name": "my-dataset" }, "type": "DB_TABLE", "name": "my-dataset", "physicalName": "public.mytable", "createdAt": "2019-05-09T19:49:24.201361Z", "updatedAt": "2019-05-09T19:49:24.201361Z", "namespace": "my-namespace", "sourceName": "my-source", "fields": [ { "name'": "a", "type": "INTEGER", "tags": [] }, { "name'": "b", "type": "TIMESTAMP", "tags": [] }, { "name'": "c", "type": "INTEGER", "tags": [] }, { "name'": "d", "type": "INTEGER", "tags": [] } ], "tags": [], "lastModifiedAt": null, "description": "My first dataset!", "facets": {}, "currentVersion": "b1d626a2-6d3a-475e-9ecf-943176d4a8c6" } ``` -------------------------------- ### Retrieve a Run by ID (C# Example) Source: https://marquezproject.ai/docs/api/get-run This C# snippet utilizes the 'RestSharp' library to make a GET request for a specific run. It configures the client, request, and handles the response. ```csharp using RestSharp; using System; public class MarquezClient { public static void Main(string[] args) { string baseURL = "http://localhost:5000/api/v1"; string runId = "YOUR_RUN_ID"; // Replace with the actual run ID var client = new RestClient(baseURL); var request = new RestRequest($"/jobs/runs/{runId}", Method.Get); request.AddHeader("Accept", "application/json"); var response = client.Execute(request); if (response.IsSuccessful) { Console.WriteLine(response.Content); } else { Console.WriteLine($"Error: {response.StatusCode} - {response.StatusDescription}"); } } } ``` -------------------------------- ### Retrieve a Run by ID (Node.js Example) Source: https://marquezproject.ai/docs/api/get-run This Node.js snippet uses the 'axios' library to fetch run details. It makes a GET request to the API endpoint and processes the JSON response. ```javascript const axios = require('axios'); const baseURL = 'http://localhost:5000/api/v1'; const runId = 'YOUR_RUN_ID'; // Replace with the actual run ID const url = `${baseURL}/jobs/runs/${runId}`; axios.get(url, { headers: { 'Accept': 'application/json' } }) .then(response => { console.log(response.data); }) .catch(error => { console.error(`Error: ${error.response.status} - ${error.response.statusText}`); }); ``` -------------------------------- ### Tag Dataset - Ruby Net::HTTP Example Source: https://marquezproject.ai/docs/api/add-tag-to-dataset Example of how to tag a dataset using Ruby's Net::HTTP library. This code shows how to set up the request and send it to the API. ```ruby require 'net/http' require 'uri' uri = URI.parse('http://localhost:5000/api/v1/namespaces/:namespace/datasets/:dataset/tags/:tag') request = Net::HTTP::Post.new(uri) request['Accept'] = 'application/json' response = Net::HTTP.start(uri.hostname, uri.port) do |http| http.request(request) end puts response.code ```