NOMAD Onboarding Workshop

https://fairmat-nfdi.github.io/nomad-onboarding-workshop
Admin
This workshop introduces the NOMAD platform, its functionalities, and practical use cases in...
Tokens:42,370
Snippets:252
Trust Score:8.3
Update:11 months ago
Show doc for...
Tokens:
Raw
### Install pymatviz Package

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_x_examples/M4_x_1_perovskites_crabnet_ml/M4_x_2_perovskites_crabnet_load_and_clean_data

Installs the `pymatviz` package, which is used for data visualization. It provides two methods: direct installation via pip and installation from the developer's GitHub repository for the latest version.

```bash
!pip install pymatviz

```

```bash
!pip install git+https://github.com/janosh/pymatviz

```

--------------------------------

### Install Libraries with Pip

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_x_examples/M4_x_1_perovskites_crabnet_ml/M4_x_1_perovskites_crabnet_retrieve_data_api

Installs the PyTorch, CrabNet, and Pandas libraries using pip. This is a prerequisite for running the rest of the example.

```shell
! pip install torch
! pip install crabnet
! pip install pandas

```

--------------------------------

### Example Response - JSON

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_3_api_helpers/M4_2_3_1_helper_explore

An example response from the API after executing the example request body.  It shows the structure of the returned JSON, including the location of the entry_id within the data.

```json
{
  "owner": "visible",
  "query": {
    "results.material.elements:all": [
      "Ti",
      "O"
    ]
  },
  "pagination": {
    "page_size": 1,
    "order_by": "upload_create_time",
    "order": "desc",
    "total": 41544,
    "next_page_after_value": "1717097142713:zIaQ0vsZ0bx4tE8n6L7pDK7SyM-k"
  },
  "required": {
    "include": [
      "entry_id",
      "upload_create_time"
    ]
  },
  "data": [
    {
      "upload_create_time": "2024-05-30T19:25:42.713000+00:00",
      "entry_id": "zIaQ0vsZ0bx4tE8n6L7pDK7SyM-k"
    }
  ]
}
```

--------------------------------

### NOMAD API Full JSON Response Example

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

A complete example of a JSON response from the NOMAD API, illustrating all major fields and their typical content. This response includes details about the query, pagination, and the actual data returned.

```json
{
  "owner": "public",
  "query": {
    "name": "results.material.elements",
    "value": {
      "all": [
        "Ti",
        "O"
      ]
    }
  },
  "pagination": {
    "page_size": 1,
    "order_by": "entry_id",
    "order": "asc",
    "total": 41544,
    "next_page_after_value": "---5IlI80yY5VWEpBMT_H-TBHOiH"
  },
  "required": {
    "include": [
      "entry_id"
    ]
  },
  "data": [
    {
      "entry_id": "---5IlI80yY5VWEpBMT_H-TBHOiH"
    }
  ]
}

```

--------------------------------

### General API Workflow

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_1_getting_started/M4_1_3_api_basics

Explanation of the typical request-response cycle in API interactions.

```APIDOC
## General API Workflow

### Description

This outlines the standard workflow for interacting with APIs, from request creation to response handling.

### Workflow Steps

1.  **Request Creation**:
    *   **HTTP Method**: Specifies the action (e.g., GET, POST).
    *   **Endpoint**: The URL for the resource or service.
    *   **Headers**: Metadata like authentication tokens or content type.
    *   **Body**: Data for creating or updating resources.

2.  **Server Processing**:
    *   The server validates the incoming request.
    *   If valid, it interacts with resources (e.g., database) and prepares a response.
    *   If invalid, it prepares an error response (e.g., 400 Bad Request).

3.  **Response Sending**:
    *   **Status Code**: Indicates success (e.g., 200) or failure (e.g., 404).
    *   **Headers**: Metadata about the response.
    *   **Body**: Contains data or error details, often in JSON or XML format.
```

--------------------------------

### Example Request Body - JSON

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_3_api_helpers/M4_2_3_1_helper_explore

An example request body used to retrieve specific data (entry_id) from the API.  This demonstrates how to filter the response and reduce page size for faster results. This body includes owner, query, pagination, and required fields.

```json
{
  "owner": "visible",
  "query": {
    "results.material.elements:all": [
      "Ti",
      "O"
    ]
  },
  "pagination": {
    "order_by": "upload_create_time",
    "order": "desc",
    "page_size": 1
  },
  "required": {
    "include": [
      "entry_id"
    ]
  }
}
```

--------------------------------

### NOMAD API Pagination Section Example

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

An example of the 'pagination' section in a NOMAD API JSON response. This illustrates the parameters used for managing and navigating through search results, such as page size, ordering, and total count.

```json
{
  "pagination": {
    "page_size": 1,
    "order_by": "entry_id",
    "order": "asc",
    "total": 41544,
    "next_page_after_value": "---5IlI80yY5VWEpBMT_H-TBHOiH"
  }
}

```

--------------------------------

### Common HTTP Methods

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_1_getting_started/M4_1_3_api_basics

Overview of the most frequently used HTTP methods for interacting with API resources.

```APIDOC
## Common HTTP Methods

### Description

This section describes the common HTTP methods used for API interactions.

### Methods

*   **GET**: Retrieves data from a server. Commonly used to fetch webpages or resources.
*   **POST**: Sends data to the server to create a new resource.
*   **DELETE**: Removes a resource from the server at a specified URL.
*   **PUT**: Updates an existing resource or creates a new one if it doesn’t exist.
```

--------------------------------

### Show example of API data formats JSON and XML

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_1_getting_started/M4_1_3_api_basics

These code examples illustrate typical data representations used in APIs. JSON and XML snippets provide a simple contact object with fields for name, role, email, and phone. They can be used as payloads for API requests or responses.

```JSON
{
  "name": "Max Mustermann",
  "role": "Lab Technician",
  "email": "max.mustermann@uni-something.de",
  "phone": "+49 123 456789"
}
```

```XML
<contact>
  <name>Max Mustermann</name>
  <role>Lab Technician</role>
  <email>max.mustermann@uni-something.de</email>
  <phone>+49 123 456789</phone>
</contact>
```

--------------------------------

### API Endpoints

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_1_getting_started/M4_1_3_api_basics

Understanding what API endpoints are and their role in accessing API resources.

```APIDOC
## API Endpoints

### Description

An API endpoint is a specific URL that serves as the entry point for interacting with a particular resource or service provided by an API. Endpoints define where requests should be sent and what actions can be performed.

### Example

In the NOMAD API, each resource has specific endpoints that allow for data retrieval, creation, updates, or deletion. The NOMAD API Dashboard highlights these endpoints and their supported request types.
```

--------------------------------

### NOMAD API Query Section Example

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

An example of the 'query' section within a NOMAD API JSON response. This demonstrates how the original search query is represented in a normalized format, including the 'name' and 'value' of the query parameters.

```json
{
  "query": {
    "name": "results.material.elements",
    "value": {
      "all": [
        "Ti",
        "O"
      ]
    }
  }
}

```

--------------------------------

### HTTP Status Codes

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_1_getting_started/M4_1_3_api_basics

Explanation of common HTTP status codes and their meanings within the context of the NOMAD API.

```APIDOC
## HTTP Status Codes

### Description

HTTP status codes indicate the outcome of an API request, helping to diagnose and address issues.

### Common Status Codes (NOMAD API)

| Response Code | Description         | Common Causes               | Suggested Action              |
|---------------|---------------------|-----------------------------|-------------------------------|
| **200**       | Successful Response | Correct request execution   | None needed                   |
| **400**       | Bad Request         | Syntax errors, invalid args | Fix syntax and parameters     |
| **401**       | Unauthorized        | Invalid credentials         | Verify credentials            |
| **404**       | Not Found           | Incorrect URL, missing res. | Check URL and resource        |
| **422**       | Validation Error    | Semantic errors in data     | Adjust data to API doc.       |
```

--------------------------------

### GET /entries/query Response Structure

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

This snippet outlines the typical JSON response structure for a query to the /entries/query endpoint, detailing the purpose of each top-level field.

```APIDOC
## GET /entries/query Response Structure

### Description
This outlines the typical JSON response structure for a query to the `/entries/query` endpoint. It details the purpose of each top-level field, including `owner`, `query`, `pagination`, `required`, and `data`.

### Method
GET

### Endpoint
/entries/query

### Response
#### Success Response (200)
- **owner** (string) - Indicates the visibility and ownership of the queried data (e.g., `admin`, `all`, `public`, `shared`, `staging`, `user`, `visible`).
- **query** (object) - A normalized representation of the search parameters sent in the request.
  - **name** (string) - The name of the query parameter.
  - **value** (object) - The value associated with the query parameter.
- **pagination** (object) - Manages how results are delivered.
  - **page_size** (integer) - The number of results returned per page.
  - **order_by** (string) - The criteria used to order the entries (e.g., `entry_id`).
  - **order** (string) - The sorting direction (`asc` or `desc`).
  - **total** (integer) - The total number of entries matching the query.
  - **next_page_after_value** (string) - A cursor for fetching subsequent pages of results.
- **required** (object) - Details about elements or parameters explicitly included or excluded in the response.
  - **include** (array) - A list of fields that were requested to be included.
- **data** (array) - The actual data matching the query.
  - **entry_id** (string) - The unique identifier for an entry.

#### Response Example
```json
{
  "owner": "public",
  "query": {
    "name": "results.material.elements",
    "value": {
      "all": [
        "Ti",
        "O"
      ]
    }
  },
  "pagination": {
    "page_size": 1,
    "order_by": "entry_id",
    "order": "asc",
    "total": 41544,
    "next_page_after_value": "---5IlI80yY5VWEpBMT_H-TBHOiH"
  },
  "required": {
    "include": [
      "entry_id"
    ]
  },
  "data": [
    {
      "entry_id": "---5IlI80yY5VWEpBMT_H-TBHOiH"
    }
  ]
}
```
```

--------------------------------

### Get DataFrame Shape with Pandas

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_x_examples/M4_x_1_perovskites_crabnet_ml/M4_x_2_perovskites_crabnet_load_and_clean_data

Retrieves and displays the dimensions (number of rows and columns) of the `df_grouped_formula` DataFrame. This provides a concise summary of the dataset size after data processing.

```python
df_grouped_formula.shape

```

--------------------------------

### Create a New Dataset via NOMAD API (Python)

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_3_upload_data/M4_3_2_upload

This code example shows how to create a new dataset using the NOMAD API. It requires a previously obtained access token and sends a POST request to the '/datasets/' endpoint with the desired dataset name in the JSON payload. The response contains details of the created dataset.

```python
base_url ='https://nomad-lab.eu/prod/v1/api/v1'
endpoint = '/datasets/'
my_dataset ='replace_the_dataset_name_here'
response = requests.post( base_url + endpoint, headers={'Authorization': f'Bearer {token}', 'Accept': 'application/json'},
            json={"dataset_name": my_dataset}
            )

```

--------------------------------

### Authenticate with NOMAD API using Python Package

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_3_upload_data/M4_3_1_authentication

This Python snippet illustrates how to authenticate with the NOMAD API using the installed NOMAD Python package. The Auth implementation handles credentials, and the example shows how to use it to fetch user uploads.

```python
import requests
from nomad.client import Auth

response = requests.get(
    'https://nomad-lab.eu/prod/v1/api/v1/uploads',
    auth=Auth(user='myname or email', password='mypassword')
)
uploads = response.json()['data']
```

--------------------------------

### Get Access Token

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_3_upload_data/M4_3_1_authentication

Obtain a short-lived access token for API authentication by sending a GET request with username and password.

```APIDOC
## GET /auth/token

### Description
Obtains a short-lived access token for authenticating API requests. This is necessary for operations requiring user identity, such as uploading or editing data.

### Method
GET

### Endpoint
`/auth/token`

### Parameters
#### Query Parameters
- **username** (string) - Required - The user's NOMAD username or email.
- **password** (string) - Required - The user's NOMAD password.

### Request Example
```python
import requests

response = requests.get(
    'https://nomad-lab.eu/prod/v1/api/v1/auth/token',
    params=dict(username='myname', password='mypassword')
)

token = response.json()['access_token']
print(f"Access Token: {token}")
```

### Response
#### Success Response (200)
- **access_token** (string) - The obtained access token.
- **token_type** (string) - The type of token, typically 'Bearer'.
- **expires_in** (integer) - The token's expiration time in seconds.

#### Response Example
```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "Bearer",
  "expires_in": 3600
}
```
```

--------------------------------

### Get NOMAD API Access Token (Python)

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_3_upload_data/M4_3_2_upload

This snippet demonstrates how to obtain an access token from the NOMAD API's '/auth/token' endpoint. It requires user credentials and uses the 'requests' library to send a GET request. The returned JSON contains the 'access_token'.

```python
import requests
import json

response = requests.get(
    'https://nomad-lab.eu/prod/v1/api/v1/auth/token', params=dict(username='my_username', password='my_password'))

token = response.json()['access_token']

```

--------------------------------

### POST /entries/query

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

This endpoint creates a search query to retrieve NOMAD entries matching specified criteria. It returns the requested fields for the matching entries, such as entry identifiers.

```APIDOC
## POST /entries/query\n\n### Description\nSearch NOMAD entries based on criteria such as chemical composition. Returns matching entry identifiers.\n\n### Method\nPOST\n\n### Endpoint\nhttp://nomad-lab.eu/prod/v1/api/v1/entries/query\n\n### Parameters\n#### Path Parameters\n_None_\n\n#### Query Parameters\n_None_\n\n#### Request Body\n- **query** (object) - Required - Defines search filters.\n- **pagination** (object) - Optional - Controls paging.\n- **required** (object) - Optional - Specifies fields to include in response.\n\n### Request Example\n{\n  "query": {\n    "results.material.elements": {\n      "all": ["Ti", "O"]\n    }\n  },\n  "pagination": {\n    "page_size": 1\n  },\n  "required": {\n    "include": ["entry_id"]\n  }\n}\n\n### Response\n#### Success Response (200)\n- **entry_id** (string) - Identifier of the matching entry.\n\n### Response Example\n{\n  "data": [\n    {\n      "entry_id": "some-id"\n    }\n  ],\n  "meta": {\n    "total": 1\n  }\n}
```

--------------------------------

### Send POST request to NOMAD API using Python requests

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

The snippet shows how to construct headers and a JSON payload, perform a POST request with the requests library, and pretty‑print the JSON response. It requires the requests and json modules and works in any Python environment. The output includes entry identifiers matching the query criteria.

```python
import requests\nimport json\n\nurl = "http://nomad-lab.eu/prod/v1/api/v1/entries/query"\n\nheaders = {\n    "Content-Type": "application/json",\n    "Accept": "application/json"\n}\n\nrequest_data = {\n    "query": {\n        "results.material.elements": {\n            "all": ["Ti", "O"]\n        }\n    },\n    "pagination": {\n        "page_size": 1\n    },\n    "required": {\n        "include": ["entry_id"]\n    }\n}\n\nresponse = requests.post(url, headers=headers, data=json.dumps(request_data))\n\nformatted_response = json.dumps(response.json(), indent=2)\nprint(formatted_response)
```

--------------------------------

### POST /entries/query

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_3_api_helpers/M4_2_3_1_helper_explore

Query entries endpoint for retrieving entry data from the NOMAD repository. Supports filtering by material elements, pagination, and field selection through a POST request with JSON body.

```APIDOC
## POST /entries/query

### Description
Query entries endpoint for retrieving entry data from the NOMAD repository using POST method with JSON body

### Method
POST

### Endpoint
/entries/metadata/entries/query

### Parameters
#### Request Body
- **owner** (string) - Required - Visibility of entries to query (e.g., "visible")
- **query** (object) - Optional - Query object for filtering results
- **pagination** (object) - Optional - Pagination settings
  - **order_by** (string) - Optional - Field to order results by (e.g., "upload_create_time")
  - **order** (string) - Optional - Order direction ("desc", "asc")
  - **page_size** (integer) - Optional - Number of results per page
- **required** (object) - Optional - Field selection for response
  - **include** (array) - Optional - List of fields to include in response

### Request Example
{
  "owner": "visible",
  "query": {
    "results.material.elements:all": [
      "Ti",
      "O"
    ]
  },
  "pagination": {
    "order_by": "upload_create_time",
    "order": "desc",
    "page_size": 1
  },
  "required": {
    "include": [
      "entry_id"
    ]
  }
}

### Response
#### Success Response (200)
- **data** (array) - Array of entry objects matching the query
- **pagination** (object) - Pagination information including total count and next page token
- **required** (object) - Echo of requested fields
- **query** (object) - Echo of query parameters
- **owner** (string) - Echo of owner parameter

#### Response Example
{
  "owner": "visible",
  "query": {
    "results.material.elements:all": [
      "Ti",
      "O"
    ]
  },
  "pagination": {
    "page_size": 1,
    "order_by": "upload_create_time",
    "order": "desc",
    "total": 41544,
    "next_page_after_value": "1717097142713:zIaQ0vsZ0bx4tE8n6L7pDK7SyM-k"
  },
  "required": {
    "include": [
      "entry_id",
      "upload_create_time"
    ]
  },
  "data": [
    {
      "upload_create_time": "2024-05-30T19:25:42.713000+00:00",
      "entry_id": "zIaQ0vsZ0bx4tE8n6L7pDK7SyM-k"
    }
  ]
}
```

--------------------------------

### Get App Token

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_3_upload_data/M4_3_1_authentication

Request an app token with a user-defined expiration time for longer sessions, using an existing access token for authorization.

```APIDOC
## GET /auth/app_token

### Description
Requests an app token with a user-defined expiration time. This is useful for longer-term access needs where the default access token's short lifespan is insufficient. Requires an existing valid access token for authorization.

### Method
GET

### Endpoint
`/auth/app_token`

### Parameters
#### Query Parameters
- **expires_in** (integer) - Optional - The desired expiration time for the app token in seconds. Defaults to a system-defined value if not provided.

#### Headers
- **Authorization** (string) - Required - The authorization token in the format `Bearer {access_token}`.

### Request Example
```python
import requests

# Assume 'token' holds a valid access token obtained previously
token = "your_access_token"

response = requests.get(
    'https://nomad-lab.eu/prod/v1/api/v1/auth/app_token?expires_in=86400',
    headers={'Authorization': f'Bearer {token}'}
)

app_token = response.json()['access_token']
print(f"App Token: {app_token}")
```

### Response
#### Success Response (200)
- **access_token** (string) - The obtained app token.
- **token_type** (string) - The type of token, typically 'Bearer'.
- **expires_in** (integer) - The token's expiration time in seconds.

#### Response Example
```json
{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "Bearer",
  "expires_in": 86400
}
```
```

--------------------------------

### NOMAD API Simplified JSON Response Structure

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_2_retrieve_data/M4_2_1_example_api_explained

A simplified JSON structure of a NOMAD API response, highlighting the main sections: 'owner', 'query', 'pagination', 'required', and 'data'. This provides a high-level overview of the response organization.

```json
{
  "owner": "public",
  "query": {...},
  "pagination": {...},
  "required": {...},
  "data": [{"entry_id": "---5IlI80yY5VWEpBMT_H-TBHOiH"}]
}

```

--------------------------------

### Statistical Summary of Dataset with Pandas

Source: https://fairmat-nfdi.github.io/nomad-onboarding-workshop/Module4/M4_x_examples/M4_x_1_perovskites_crabnet_ml/M4_x_2_perovskites_crabnet_load_and_clean_data

Generates and displays a statistical summary of the DataFrame, providing count, mean, standard deviation, min, max, and quartile values for numerical columns. This helps in understanding the distribution and identifying potential outliers.

```python
df.describe()

```
Tokens:
Raw