Hagrid

https://freshworks-oss.github.io/hagrid-oss
Admin
Hagrid is a Java-based connector development framework provided as an SDK, designed to simplify the...
Tokens:32,450
Snippets:434
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Setup Step Method

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods?q=

The setup method acts as a pre-hook, allowing for logging or analytics before the main execution of a step begins.

```java
@Override
    public void setup(ImmutableMap<String, String> baggageMap, JsonNode... parentJsonObject) throws StepFailedException {
        analyticsService.infoEvent("METHOD_CALLED", "name", "setup");
    }
```

--------------------------------

### Pre-execution Setup

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods

The `setup` method acts as a pre-hook for logging or analytics before the main execution. It receives baggage information and parent JSON objects.

```java
@Override
    public void setup(ImmutableMap<String, String> baggageMap, JsonNode... parentJsonObject) throws StepFailedException {
        analyticsService.infoEvent("METHOD_CALLED", "name", "setup");
    }

```

--------------------------------

### setup

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods?q=

A pre-hook method called by Hagrid to allow developers to perform logging or analytics before the main execution.

```APIDOC
## setup

### Description
This method is like a `pre-hook`, `Hagrid` called this method to allow dev to perform any kind of `logging`, `analytics` before it actually starts executing this method.

### Method
setup

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```java
@Override
public void setup(ImmutableMap<String, String> baggageMap, JsonNode... parentJsonObject) throws StepFailedException {
    analyticsService.infoEvent("METHOD_CALLED", "name", "setup");
}
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### StartSync

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods?q=

Called to get the initial URL for fetching data for the current step. Developers create and configure an `HttpRequestResponse` object here.

```APIDOC
## StartSync

### Description
This method, will be called to get the `first-url` to call to get the data for this step. Here you create the object of `HttpRequestResponse` and set the `HttpRequest` object. Hagrid will make a call to third-party and set the `HttpResponse` in the same `HttpRequestResponse`.

### Method
startSync

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```java
@Override
public Optional startSync(JsonNode... parentJsonObject) throws StepFailedException {
    analyticsService.infoEvent("METHOD_CALLED", "name", "startSync");
    try{
        HttpRequestResponse httpRequestResponse = new HttpRequestResponse();
        HttpRequest httpRequest = new HttpRequest();
        httpRequest.initGet("https://l3rtckyana.execute-api.us-east-1.amazonaws.com/performance-testing/user?how_many=" + numberOfUsersEachPage);
        httpRequestResponse.setRequest(httpRequest);
        return Optional.fromNullable(httpRequestResponse);
    }
    catch (Exception e){
        e.printStackTrace();
        return Optional.absent();
    }
}
```

### Response
#### Success Response (200)
- **Optional** - Contains `HttpRequestResponse` if successful, otherwise absent.

#### Response Example
```json
{
  "example": "Optional.fromNullable(httpRequestResponse)"
}
```
```

--------------------------------

### Example JSON Response Body

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/steps

This is an example of a JSON response body from a third-party service.

```json
{
    "post_id" : "1234",
    "post_title" : " Having good vacations",
    "post_description" : " vacation in India",
    "total_likes" : 100
    "total_comments" : 50
  }
```

--------------------------------

### Run Hagrid with Docker Setup

Source: https://freshworks-oss.github.io/hagrid-oss/setup/playground

Execute Hagrid locally, connecting to Django, Grafana, and Prometheus containers managed by Docker. This setup allows for testing Hagrid's interaction with simulated third-party APIs and monitoring its performance metrics.

```bash
docker-compose up
```

--------------------------------

### StartSync Method for Initial Request

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods?q=

This method is called to obtain the initial URL for data fetching. It involves creating and configuring an `HttpRequestResponse` object for a GET request.

```java
@Override
    public Optional startSync(JsonNode... parentJsonObject) throws StepFailedException {
        analyticsService.infoEvent("METHOD_CALLED", "name", "startSync");
        try{
            HttpRequestResponse httpRequestResponse = new HttpRequestResponse();
            HttpRequest httpRequest = new HttpRequest();
            httpRequest.initGet("https://l3rtckyana.execute-api.us-east-1.amazonaws.com/performance-testing/user?how_many=" + numberOfUsersEachPage);
            httpRequestResponse.setRequest(httpRequest);
            return Optional.fromNullable(httpRequestResponse);
        }
        catch (Exception e){
            e.printStackTrace();
            return Optional.absent();
        }
    }
```

--------------------------------

### Run Hagrid for Testing

Source: https://freshworks-oss.github.io/hagrid-oss/setup/playground?q=

Execute this command to run Hagrid locally. This setup allows Hagrid to connect with the Dockerized services (Django, Grafana, Prometheus) for testing and monitoring.

```bash
go run main.go
```

--------------------------------

### Define an Inner Join Asset for User Usage

Source: https://freshworks-oss.github.io/hagrid-oss/services/processor_services

Example of defining an asset that requires two beans (User and Usage) joined on specific fields. Demonstrates INNER_JOIN.

```java
@Setter
@NoArgsConstructor
@JsonInclude(JsonInclude.Include.NON_EMPTY)
@JsonIgnoreProperties(ignoreUnknown = true)

@FreshJoin(rightClass = com.freshworks.hagrid.beans.User.class, uniqueJoinName = "user_usage_join",join_type = FreshJoin.JOIN_TYPE.INNER_JOIN,
        onFieldList = {
                @FreshJoin.OnField(rightClassFieldName ="id", leftClassFieldName = "user_id", leftClass =  com.freshworks.hagrid.beans.Usage.class),
        }
)

public class UserUsage extends AbstractAsset {

    String user_id;
    String usageValue;
    String userName;


    public void setUser_id(User user, Usage usage){
        this.user_id = user.getUserId();
        this.usageValue = usage.getUsage();
        this.userName = user.getUserName();

    }
}

```

--------------------------------

### Add Attributes to Assets

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/assets?q=

Use the transform method to add new attributes to assets. This example shows setting the created_at timestamp.

```java
@Override
public void transform() {

    this.created_at = new Date();
}

```

--------------------------------

### Example JSON Data Structure

Source: https://freshworks-oss.github.io/hagrid-oss/pages/beans/beans

Illustrates a sample JSON response structure from a third-party system, containing metadata and a list of user data. Hagrid will selectively parse fields defined in the corresponding bean class.

```json
{
  "meta": {
    "token" : "some_token_here",
    "date" : "today_date"
  },
  "data" : [{
      "user_name" : "Amit Aggarwal",
      "user_id" : "some_fb_id_123455",
      "address" : "bangalore",
      "friends_count" : 100,
      "last_login" : "1 year back"
  },{
    "user_name" : "Praveen",
    "user_id" : "some_fb_id_098987",
    "address" : "Kerala",
    "friends_count" : 100,
    "last_login" : "2 year back"
  }]
}

```

--------------------------------

### Consume All Assets After Sync Completion

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/consumer?q=

Use `consumerService.getAssetByAssetType` to fetch all assets of a specific type once the synchronization process is complete. Ensure `syncStatusService.waitUntilSyncIsInProgress()` is called to wait for the sync to start.

```java
SyncServiceContainer syncServiceContainer = syncService.startSync(ParentStep.class, namespace,  map);
ConsumerService consumerService = syncServiceContainer.getBean(ConsumerService.class);
SyncStatusService syncStatusService = syncServiceContainer.getBean(SyncStatusService.class);
syncStatusService.waitUntilSyncIsInProgress();
analyticsService.meterGauge("HAGRID_STATUS", 1);
List<FbComment> commentList = consumerService.getAssetByAssetType(FbComment.class);
```

--------------------------------

### Checkout Hagrid Initializer Branch

Source: https://freshworks-oss.github.io/hagrid-oss/setup/playground

Switch to the 'hagrid/initializer' branch of the Hagrid repository to get the latest release's initializer code. This branch contains fully functional code that uses the specific Hagrid version for easy feature exploration.

```bash
git checkout hagrid/initializer
```

--------------------------------

### Fetch InfraService and InfraDbList

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/steps

Demonstrates fetching `InfraBeanConfiguration` and subsequently `InfraService` and `InfraDbList` from the `SyncServiceContainer` to manage infrastructure data.

```java
@Slf4j
@FreshHierarchy(parentClass = FbPost.class, rateLimit = 800, duration = 1, ignore = false)
@Component
@Scope("prototype")
public class FbComment extends AbstractStep {

    @Override
    public void configure(SyncServiceContainer syncServiceContainer) {
        InfraBeanConfiguration infraBeanConfiguration = syncServiceContainer.getBean(InfraBeanConfiguration.class);
        InfraService infraService = infraBeanConfiguration.getInfraService();
        InfraDbList infraDbList = infraService.getInfraDbList("list_name");
    }

}
```

--------------------------------

### Configure Step Initialization

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods

The `configure` method is called upon step object creation. Use the `SyncServiceContainer` to access Hagrid services.

```java
@Override
    public void configure(SyncServiceContainer syncServiceContainer){
        // Here use syncService container to get InfraService, AnalyticsService, SyncStatusService etc. 
    }

```

--------------------------------

### Modify Attributes of Bean

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/beans

Update existing attributes of a bean using the `transform` method. This example converts the `name` attribute to lowercase.

```java
// Modify attributes
@Override
public void transform() {

    this.name = this.name.toLower();
}
```

--------------------------------

### Initiate Data Synchronization

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods

The `startSync` method is called to obtain the initial URL for data fetching. It involves creating and configuring an `HttpRequest` object within `HttpRequestResponse`.

```java
@Override
    public Optional startSync(JsonNode... parentJsonObject) throws StepFailedException {
        analyticsService.infoEvent("METHOD_CALLED", "name", "startSync");
        try{
            HttpRequestResponse httpRequestResponse = new HttpRequestResponse();
            HttpRequest httpRequest = new HttpRequest();
            httpRequest.initGet("https://l3rtckyana.execute-api.us-east-1.amazonaws.com/performance-testing/user?how_many=" + numberOfUsersEachPage);
            httpRequestResponse.setRequest(httpRequest);
            return Optional.fromNullable(httpRequestResponse);
        }
        catch (Exception e){
            e.printStackTrace();
            return Optional.absent();
        }
    }

```

--------------------------------

### Define a Facebook Comment Asset

Source: https://freshworks-oss.github.io/hagrid-oss/services/processor_services

Example of defining a simple asset that extends AbstractAsset and depends on a single bean. Used for Noop Assets.

```java
@NoArgsConstructor
@Getter
@Setter
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonInclude(JsonInclude.Include.NON_EMPTY)

public class FbComment extends AbstractAsset {

    String comment_id;
    String comment_title;
    String comment_text;

    public void setBatchFromBean(FbComment comment){

        comment_id = comment.getComment_id();
        comment_title = comment.getComment_title();
        comment_text = comment.getComment_text();
    }

    @Override
    public void transform() {
//        System.out.println("Creating comment asset");
    }

    @Override
    public Object getUniqueIdentifier() {
        return null;
    }
}

```

--------------------------------

### Delete Attributes from Bean

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/beans

Remove unwanted attributes from a bean by setting them to `null` within the `transform` method. This example demonstrates deleting `unwantedAttribute`.

```java
// Delete unwanted attributes
@Override
public void transform() {

    this.unwantedAttribute = null;
}
```

--------------------------------

### Example Bean Class Definition

Source: https://freshworks-oss.github.io/hagrid-oss/pages/beans/beans

Defines a bean class `FbUser` that extends `AbstractBean`. It specifies the attributes to be retained and others to be discarded during data processing.

```java
@Getter
@Setter
@NoArgsConstructor
@JsonIgnoreProperties(ignoreUnknown = true)
public class FbUser extends AbstractBean {

    String user_id;
    String user_name;
}

```

--------------------------------

### configure

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods?q=

Configure is called once the object of `Step` is created. It receives a `SyncServiceContainer` which provides access to all relevant Hagrid services.

```APIDOC
## configure

### Description
Configure is the method that will be called once the object of `Step` is created. Parameter of `configure` method is `SyncServiceContainer`. You can think of `SyncServiceContainer` like `ApplicationContext` of spring. With `SyncServiceContainer` contains instances of `all Hagrid based services` which are relevant to the current run.

### Method
configure

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```java
@Override
public void configure(SyncServiceContainer syncServiceContainer){
    // Here use syncService container to get InfraService, AnalyticsService, SyncStatusService etc. 
}
```

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Add Attributes to Bean

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/beans

Use the `transform` method inherited from `AbstractBean` to add new attributes to a bean. This example shows setting the `created_at` attribute.

```java
import java.util.Date;

// Add attributes
@Override
public void transform() {

    this.created_at = new Date();
}
```

--------------------------------

### Modify Attributes of Assets

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/assets

Update existing attributes of an asset by overriding the transform method and modifying the attribute's value, for example, converting a name to lowercase.

```java
// Modify attributes
@Override
public void transform() {

    this.name = this.name.toLower();
}

```

--------------------------------

### Configure Dynamic Rate Limit

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/steps

This snippet demonstrates how to set a dynamic rate limit for a specific step at runtime using Hagrid's FreshHierarchy annotation. It requires initializing a SyncServiceContainer and configuring the TraverseConfigService.

```java
public void some_app_method(){

    SyncServiceContainer syncServiceContainer = syncService.initSyncServiceContainer("namespace here");
    TraversConfigService traverConfigService = syncServiceContainer.getBean(TraverseConfig.class);

    traverseConfigService.setRateLimit(FbUserStep.class, 200, 20);

    syncService.startSync(syncServiceContainer, FBUser.class, baggageMap);

}


```

--------------------------------

### Configure NitriteDB for Discovery Connector

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/44

Use these keys in `hagrid.yml` to configure NitriteDB for discovery connectors. Specify the data path and database type.

```yaml
infra:
    infra_type: "nitrite"
    nitrite_data_path: "/Users/aaggarwal/Documents/hagrid-releases/hagrid-oss/hagrid-oss/database"
    nitrite_database_type: "file"
```

--------------------------------

### Accessing TraverseConfigService using SyncServiceContainer

Source: https://freshworks-oss.github.io/hagrid-oss/services/shared_services

Demonstrates how to fetch and use the TraverseConfigService to set dynamic rate limits for a specific user class.

```java
TraverseConfigService traverseConfigService = syncServiceContainer.getBean(TraverseConfigService.class);
traverseConfigService.setStepRateLimit(FbUser.class, 200, 20);
```

--------------------------------

### Define a Simple Asset (FbComment)

Source: https://freshworks-oss.github.io/hagrid-oss/services/processor_services?q=

This Java class defines a simple asset 'FbComment' that extends 'AbstractAsset'. It includes fields for comment details and a method to set batch data from a bean. Use this for assets that depend on a single bean.

```java
import lombok.Getter;
import lombok.Setter;
import lombok.NoArgsConstructor;
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonInclude;

@NoArgsConstructor
@Getter
@Setter
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonInclude(JsonInclude.Include.NON_EMPTY)

public class FbComment extends AbstractAsset {

    String comment_id;
    String comment_title;
    String comment_text;

    public void setBatchFromBean(FbComment comment){

        comment_id = comment.getComment_id();
        comment_title = comment.getComment_title();
        comment_text = comment.getComment_text();
    }

    @Override
    public void transform() {
//        System.out.println("Creating comment asset");
    }

    @Override
    public Object getUniqueIdentifier() {
        return null;
    }
}
```

--------------------------------

### Filter Beans Based on Conditions

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/beans

Implement filtering logic for beans using the `filter` method inherited from `AbstractBean`. This example returns `false` if the `email` attribute is null.

```java
@Override
public void filter() {

    if(this.email == null){
        return false;
    }
}
```

--------------------------------

### Facebook Comment Step Annotation

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/freshhierarchy_annotation

Annotates a step class to define its parent, rate limit, and duration. This example demonstrates a deeper level in the hierarchy, linking to the 'FbPost' step.

```java
@Slf4j
@FreshHierarchy(parentClass = FbPost.class, rateLimit = 800, duration = 120, ignore = false)
@Component
@Scope("prototype")
public class FbStepComment extends HttpAbstractStep {

    // Override some methods here
}
```

--------------------------------

### Stream Assets Continuously with Consumer Service

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/consumer

Use this pseudo-code to continuously consume assets as they are generated. It involves a loop that checks sync status and fetches new assets using `consumerService.streamAssetByAssetType` with a token.

```java
AssetStreamResponse.Token token = new AssetStreamResponse.Token();
token.setStart(0);
token.setStart(100);
AssetStreamResponse assetStreamResponse = null;
while(syncStatusService.getSyncStatus() == 0 ){
assetStreamResponse = consumerService.streamAssetByAssetType(FbUser.class, token);

// Here is your list of 100 assets
List<FbUser> fbUserList = assetStreamResponse.getAbstractAssetList();

// new token return by consumerService itself.
// If there are no new assets yet then this token contains start = 101 and count - 100
// you pass this token back to consumerService to fetch data
// if there is not data then list of assets will be empty
// always use streamAssetByAssetType in loop so that you can get it terminated when sync is complete.
// consumerService itself does not know anything about syncStatus, hence we need to use while loop to keep
// fetching data until sync is complete.
token = assetStreamResponse.getNextToken();
}

// Make a final fetch in case there are some more assets
// This is to avoid a race condition like below 
// Assume 
// 1. processorService generates some assets
// 2. Sync status changes to successful 
// 3. above while loop will check if sync is progress, which is not then it will go out of loop 
// how ever, there are some more assets got generated in between. Hence we have to use this final fetch. 
assetStreamResponse = consumerService.streamAssetByAssetType(FbUser.class, token);
List<FbUser> fbUserList = assetStreamResponse.getAbstractAssetList();

```

--------------------------------

### Abort Current Parent and Continue

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods

Use this to skip the current parent step and proceed with the next parent instance. Useful for skipping processing for specific items, like a user ID, and continuing with others.

```java
@Override
    public DagTraversalService.TraverseAction handleInvalidResponse(HttpRequestResponse currentRequest, JsonNode... parentJsonObject) throws URISyntaxException, StepFailedException {

        DagTraversalService.TraverseAction traverseAction = new DagTraversalService.TraverseAction();
        traverseAction.abortCurrentParentAndContinueWithNextParentInstance();
        return null;
    }
```

--------------------------------

### Facebook Post Step Annotation

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/freshhierarchy_annotation

Annotates a step class to define its parent, rate limit, and duration. This example shows a child step where the parent is another custom step class.

```java
@Slf4j
@FreshHierarchy(parentClass = FbUser.class, rateLimit = 200, duration = 30, ignore = false)
@Component
@Scope("prototype")
public class FbStepPost extends HttpAbstractStep {

    // Override some methods here
}
```

--------------------------------

### NitriteDB Configuration for Action-Based Use Case

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/44

This configuration is for connectors that perform numerous actions on third-party services, utilizing an in-memory NitriteDB instance.

```yaml
infra:
infra_type: "nitrite"
nitrite_database_type: ""

```

--------------------------------

### NitriteDB Configuration for File-Based Discovery Use Case

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/44

This configuration is used when the connector performs heavy discovery jobs and requires a file-based NitriteDB instance.

```yaml
infra:
infra_type: "nitrite"
nitrite_data_path: "/Users/aaggarwal/Documents/hagrid-releases/hagrid-oss/hagrid-oss/database"
nitrite_database_type: "file"

```

--------------------------------

### Consume Assets Continuously

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/consumer?q=

Use this snippet in a while loop to continuously fetch and process assets as they are generated. It manages tokens to retrieve new assets and terminates when the sync status indicates completion. Ensure to handle the case where no new assets are available.

```java
AssetStreamResponse.Token token = new AssetStreamResponse.Token();
            token.setStart(0);
            token.setStart(100);
AssetStreamResponse assetStreamResponse = null;
            while(syncStatusService.getSyncStatus() == 0 ){
assetStreamResponse = consumerService.streamAssetByAssetType(FbUser.class, token);

// Here is your list of 100 assets
List<FbUser> fbUserList = assetStreamResponse.getAbstractAssetList();

// new token return by consumerService itself.
// If there are no new assets yet then this token contains start = 101 and count - 100
// you pass this token back to consumerService to fetch data
// if there is not data then list of assets will be empty
// always use streamAssetByAssetType in loop so that you can get it terminated when sync is complete.
// consumerService itself does not know anything about syncStatus, hence we need to use while loop to keep
// fetching data until sync is complete.
token = assetStreamResponse.getNextToken();
            }

// Make a final fetch in case there are some more assets
// This is to avoid a race condition like below 
// Assume 
// 1. processorService generates some assets
// 2. Sync status changes to successful 
// 3. above while loop will check if sync is progress, which is not then it will go out of loop 
// how ever, there are some more assets got generated in between. Hence we have to use this final fetch. 
assetStreamResponse = consumerService.streamAssetByAssetType(FbUser.class, token);
List<FbUser> fbUserList = assetStreamResponse.getAbstractAssetList();

```

--------------------------------

### Get Next Sync Request

Source: https://freshworks-oss.github.io/hagrid-oss/pages/steps/step_methods

Constructs and returns the next HTTP request for synchronization when `isSyncComplete` returns `false`. This method is used to implement pagination by specifying the URL for the next data fetch. It receives the previous response and parent data for context.

```java
@Override
    public Optional getNextSyncRequest(HttpRequestResponse currentRequest, JsonNode... parentJsonObject) throws StepFailedException {
        try{
            analyticsService.infoEvent("METHOD_CALLED", "name", "getNextSyncRequest");
            analyticsService.infoEvent("THIRD_PARTY_API_CALLED");

            HttpRequestResponse httpRequestResponse = new HttpRequestResponse();
            HttpRequest httpRequest = new HttpRequest();
            httpRequest.initGet("https://l3rtckyana.execute-api.us-east-1.amazonaws.com/performance-testing/user?has_next=true&how_many=100");
            httpRequestResponse.setRequest(httpRequest);

            count = count + 1;
//            Thread.sleep(waitBetweenPaginationInMs);
            return Optional.fromNullable(httpRequestResponse);
        }
        catch (Exception e){
            e.printStackTrace();
        }

        return null;
    }
```

--------------------------------

### Define a Combined Asset with FreshJoin Annotation

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/18

Use this annotation to define how two assets are joined. Specify the classes and the fields to be used for joining.

```java
@FreshJoin('rightClass'= AssetA.class, 'rightClassFieldName' = 'id', 'leftClass' = AssetB.class, 'leftClassFieldName' = 'user_id')
  public class combinedAsset extends AbstractAsset{

     String name;
     String company;

     public void setFromAsset(AssetA assetA){
      this.name = assetA.getName();  
     }

     public void setFromAsset(AssetB assetB){
      this.company = assetB.getCompanyName();
     }
  }
```

--------------------------------

### Asset Processing Loop with Queue

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/18

This code snippet demonstrates the core logic for processing beans and assets. It clears an asset list, iterates through input beans, processes primitive assets, and then enters a loop to process non-primitive assets from the list until it's empty. It includes a check for thread interruption.

```java
abstractAssetList.clear();
            for (String bean : itemList) {

                if (Boolean.FALSE.equals(Thread.interrupted())) {

                  // Processing primitive Assets first.
                    abstractAssetList = processBeanForAsset(bean);
                   while(true) {

                        if (abstractAssetList.isEmpty()){
                            break;
                        }

                        // Processing non primitive Assets
                        processAssetForAsset(abstractAssetList.pop());

                   }

                } else {
                    // If thread is interrupted or asked to terminate then skip the list and publish
                    // whatever assets are generated
                    break;
                }
            }
```

--------------------------------

### Configure Infrastructure Type for Data Storage

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/performance

Choose between 'memory' and 'persistent' infra types based on the volume of data fetched from third-party sources. 'memory' is suitable for minimal data, while 'persistent' (using MongoDB) is for larger datasets.

```text
infra_type: memory
```

```text
infra_type: persistent
```

--------------------------------

### Configure Step Paths in Hagrid

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/steps

Specify top-level paths for steps, beans, and assets to enable Hagrid to scan all inner packages. This configuration is typically placed in a YAML file.

```yaml
---
configuration:
  connector:
    step_path: "com.freshworks.hagrid.steps"
    bean_path: "com.freshworks.hagrid.beans"
    asset_path: "com.freshworks.hagrid.assets"

```

--------------------------------

### Process Beans and Create Assets

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/18?q=

This code processes a list of beans, generating primitive assets and then iteratively creating non-primitive assets. It continues processing until the abstract asset list is empty or the thread is interrupted. Use this when you need to recursively generate assets from initial beans.

```java
abstractAssetList.clear();
            for (String bean : itemList) {

                if (Boolean.FALSE.equals(Thread.interrupted())) {

                  // Processing primitive Assets first.
                    abstractAssetList = processBeanForAsset(bean);
                   while(true) {

                        if (abstractAssetList.isEmpty()){
                            break;
                        }

                        // Processing non primitive Assets
                        processAssetForAsset(abstractAssetList.pop());

                   }

                } else {
                    // If thread is interrupted or asked to terminate then skip the list and publish
                    // whatever assets are generated
                    break;
                }
            }

```

--------------------------------

### Define Primitive Asset with setFromBean

Source: https://freshworks-oss.github.io/hagrid-oss/pages/assets/assets?q=

Use this to define a primitive asset by mapping fields from a bean. Ensure the bean parameter is present for Hagrid to recognize it as a primitive asset.

```java
@NoArgsConstructor
@Getter
@Setter
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonInclude(JsonInclude.Include.NON_EMPTY)
public class FbUserAsset extends AbstractAsset {

    public void setFromBean(com.freshworks.hagrid.beans.User userBean) {

        userId = userBean.getUser_id();
        userName = userBean.getUser_name();
    }
}
```

--------------------------------

### Consume All Assets of a Given Type

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/consumer

Fetches all assets of a specified type using `consumerService.getAssetByAssetType`. This is a basic retrieval method without any filtering.

```java
ConsumerService consumerService = syncServiceContainer1.get().getBean(ConsumerService.class);

            // Use consumer service to consume Assets fetched by Hagrid

            // Consume assets of a given type
            List<User> userAssetList = consumerService.getAssetByAssetType(User.class);
```

--------------------------------

### Create URL from Previous Response

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/steps

This snippet shows how to extract information from a previous response to construct a URL for the next page in pagination scenarios. It requires access to the current request and parent JSON objects.

```java
class FbComment extends AbstractStep{

    // Other methods are omitted for clarity 

    @Override
    public Optional getNextSyncRequest(HttpRequestResponse currentRequest, JsonNode... parentJsonObject) throws StepFailedException {
        try {
            analyticsService.infoEvent("METHOD_CALLED", "name", "getNextSyncRequest");
            analyticsService.infoEvent("THIRD_PARTY_API_CALLED");

            JsonNode postData = parentJsonObject[0];
            String postId = postData.get("post_id").asText();

            String commentResponseBodyOfLastRequest = currentRequest.getResponse().getBody();

            // Here use the postId , token from commentResponseBodyOfLastRequest and create a new URL
        }
}


```

--------------------------------

### Consume Filtered Assets with Combined Conditions

Source: https://freshworks-oss.github.io/hagrid-oss/usecases/consumer?q=

Fetch assets that satisfy multiple conditions combined with an AND operator. This requires building separate expressions for each condition and then joining them.

```java
Expression expression1 = Expression.expressionBuilder()
                    .whenAssetFieldName("$.my.package.path.of.the.asset.User.group")
                    .is()
                    .whenAssetFieldValue("HR")
                    .build();

            Expression expression2 = Expression.expressionBuilder()
                            .whenAssetFieldName("$.my.package.path.of.the.asset.User.type")
                            .is()
                            .whenAssetFieldValue("2")
                            .build();

            Expression finalExpression = Expression.expressionJoiner()
                    .whenRightExpressionIs(expression1)
                            .whenJoinerIsAnd()
                                    .whenLeftExpressionIs(expression2)
                                            .build();


            List<User> userAssetListFilterByGroupAndType = consumerService.getAssetByAssetTypeAndFilter(User.class, finalExpression);
```

--------------------------------

### Define a Primitive Asset (AssetB)

Source: https://freshworks-oss.github.io/hagrid-oss/dsr/18

This class represents a primitive asset, AssetB, which can be populated from a BeanA.

```java
public class AssetB extends AbstractAsset{

     String user_id
     String department;
     String designation;

     public void setFromBean(BeanA beanB){
      this.department = beanB.getDepartment();  
     }

     public void setFromBean(BeanA beanB){
      this.designation = beanB.getDesignation();
     }
  }
```
Tokens:
Raw