Reverse Linear Sync Engine

https://github.com/wzhudev/reverse-linear-sync-engine
Admin
A detailed study and documentation of Linear's Sync Engine (LSE), exploring its model definition,...
Tokens:17,487
Snippets:72
Trust Score:9.5
Update:1 year ago
Show doc for...
Tokens:
Raw
### Example HTTP Request for Partial Model Bootstrap

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This shows an example HTTP GET request URL generated by LSE's `loadPartialModels` function. It illustrates how `modelClass` values from `customNetworkHydration` are mapped to the `onlyModels` query parameter and how `syncGroups` are included in the request for partial data bootstrapping.

```HTTP
https://client-api.linear.app/sync/bootstrap?type=partial&noSyncPackets=true&useCFCaching=true&noCache=true&firstSyncId=3577987809&syncGroups=aa788b7b-9b76-4caa-a439-36ca3b3d6820&onlyModels=Issue,Attachment&modelsHash=4f1dabd6151ad381a502c352b677d5c4
```

--------------------------------

### HTTP GET Request for Full Database Bootstrap

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet shows the HTTP GET request URL used by the Linear Sync Engine (LSE) to initiate a full database bootstrap. It includes 'type=full' and a comma-separated list of 'onlyModels' to be loaded from the server.

```HTTP
https://client-api.linear.app/sync/bootstrap?type=full&onlyModels=WorkflowState,IssueDraft,Initiative,ProjectMilestone,ProjectStatus,TextDraft,ProjectUpdate,IssueLabel,ExternalUser,CustomView,ViewPreferences,Roadmap,RoadmapToProject,Facet,Project,Document,Organization,Template,Team,Cycle,Favorite,CalendarEvent,User,Company,IssueImport,IssueRelation,TeamKey,UserSettings,PushSubscription,Activity,ApiKey,EmailIntakeAddress,Emoji,EntityExternalLink,GitAutomationTargetBranch,GitAutomationState,Integration,IntegrationsSettings,IntegrationTemplate,NotificationSubscription,OauthClientApproval,Notification,OauthClient,OrganizationDomain,OrganizationInvite,ProjectLink,ProjectUpdateInteraction,InitiativeToProject,Subscription,TeamMembership,TimeSchedule,TriageResponsibility,Webhook,WorkflowCronJobDefinition,WorkflowDefinition,ProjectRelation,DiaryEntry,Reminder
```

--------------------------------

### Example Streamed JSON Response for Full Bootstrap

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet illustrates a partial example of the streamed JSON response from the GraphQLClient.restModelsJsonStreamGen endpoint. Each line (except the last) represents a model instance, followed by a final line containing request-specific metadata.

```JSON
{"id":"8ce3d5fe-07c2-481c-bb68-cd22dd94e7de","createdAt":"2024-07-03T11:37:04.865Z","updatedAt":"2024-07-03T11:37:04.865Z","userId":"4e8622c7-0a24-412d-bf38-156e073ab384","issueId":"01a3c1cf-7dd5-4a13-b3ab-a9d064a3e31c","events":[{"type":"issue_deleted","issueId":"01a3c1cf-7dd5-4a13-b3ab-a9d064a3e31c","issueTitle":"Load data from remote sync engine."}],"__class":"Activity"}
{"id":"ec9ec347-4f90-465c-b8bc-e41dae4e11f2","createdAt":"2024-07-03T11:37:06.944Z","updatedAt":"2024-07-03T11:37:06.944Z","userId":"4e8622c7-0a24-412d-bf38-156e073ab384","issueId":"39946254-511c-4226-914f-d1669c9e5914","events":[{"type":"issue_deleted","issueId":"39946254-511c-4226-914f-d1669c9e5914","issueTitle":"Reverse engineering Linear's Sync Engine"}],"__class":"Activity"}
// ... many lines omitted here
_metadata_={
  "method": "mongo",
  "lastSyncId": 2326713666,
  "subscribedSyncGroups": [
    "89388c30-9823-4b14-8140-4e0650fbb9eb",
    "4e8622c7-0a24-412d-bf38-156e073ab384",
    "AD619ACC-AAAA-4D84-AD23-61DDCA8319A0",
    "CDA201A7-AAAA-45C5-888B-3CE8B747D26B"
  ],
  "databaseVersion": 948,
  "returnedModelsCount": {
    "Activity": 6,
    "Cycle": 2,
    "DocumentContent": 5,
    "Favorite": 1,
    "GitAutomationState": 3,
    "Integration": 1,
    "Issue": 3,
    "IssueLabel": 4,
    "NotificationSubscription": 2,
    "Organization": 1,
    "Project": 2,
    "ProjectStatus": 5,
    "Team": 1,
    "TeamKey": 1,
    "TeamMembership": 1,
    "User": 1,
    "UserSettings": 1,
    "WorkflowState": 7,
    "Initiative": 1,
    "SyncAction": 0
  }
}
```

--------------------------------

### Example JSON Object for an Issue Model

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet provides a detailed example of a single JSON object representing an 'Issue' model instance, as received within the streamed bootstrap response. It showcases various fields like ID, timestamps, title, and relationships.

```JSON
{
  "id": "556c8983-ca05-41a8-baa6-60b6e5d771c8",
  "createdAt": "2024-01-22T01:02:41.099Z",
  "updatedAt": "2024-05-16T08:23:31.724Z",
  "number": 1,
  "title": "Welcome to Linear ðŸ‘‹",
  "priority": 1,
  "boardOrder": 0,
  "sortOrder": -84.71,
  "startedAt": "2024-05-16T08:16:57.239Z",
  "labelIds": ["30889eaf-fac5-4d4d-8085-a4c3bd80e588"],
  "teamId": "89388c30-9823-4b14-8140-4e0650fbb9eb",
  "projectId": "3e7ada3c-f833-4b9c-b325-6db37285fa11",
  "projectMilestoneId": "397b95c4-3ee2-47b0-bad1-d6b1c7003616",
  "subscriberIds": ["4e8622c7-0a24-412d-bf38-156e073ab384"],
  "previousIdentifiers": [],
  "assigneeId": "4e8622c7-0a24-412d-bf38-156e073ab384",
  "stateId": "030a7891-2ba5-4f5b-9597-b750950cd866",
  "reactionData": [],
  "__class": "Issue"
}
```

--------------------------------

### Generate GraphQL Mutation for Transaction Updates

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The `prepare` method of a transaction generates a GraphQL mutation query, stored in the `graphQLMutationPrepared` property. This example demonstrates the structure of such a query, including the mutation text, variables, and their types, used for updating an issue's assignee.

```JSON
{
  "mutationText": "issueUpdate(id: \"a3dad63b-8302-4f1f-a874-a80e6d9ed418\", input: $issueUpdateInput) { lastSyncId }",
  "variables": {
    "issueUpdateInput": {
      "assigneeId": "4e8622c7-0a24-412d-bf38-156e073ab384"
    }
  },
  "variableTypes": {
    "issueUpdateInput": "IssueUpdateInput"
  }
}
```

--------------------------------

### GraphQL Request for Batched Project and Document Creation

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This example illustrates a more complex GraphQL request where LSE batches multiple transactions, specifically creating a `Project` and `DocumentContent`, into a single mutation query. It demonstrates how `operationName` and field aliases (`o1`, `o2`) are used to manage multiple operations and their respective variables within one request.

```JSON
{
  "query": "mutation ProjectCreate_DocumentContentCreate($projectCreateInput: ProjectCreateInput!, $documentContentCreateInput: DocumentContentCreateInput!) { o1:projectCreate(input: $projectCreateInput) { lastSyncId }, o2:documentContentCreate(input: $documentContentCreateInput) { lastSyncId } }",
  "variables": {
    "projectCreateInput": {
      "id": "940e248c-2226-4b0a-a14e-8a410ccfaaa7",
      "name": "New Project",
      "description": "",
      "color": "#bec2c8",
      "statusId": "fd3884d6-b740-41d9-919c-796119d6c5ed",
      "memberIds": [],
      "sortOrder": 1005.71,
      "prioritySortOrder": 0,
      "priority": 0,
      "teamIds": ["369af3b8-7d07-426f-aaad-773eccd97202"]
    },
    "documentContentCreateInput": {
      "id": "bb75174e-1e26-46ae-94b6-67977be435c3",
      "projectId": "940e248c-2226-4b0a-a14e-8a410ccfaaa7"
    }
  },
  "operationName": "ProjectCreate_DocumentContentCreate"
}
```

--------------------------------

### LSE: Core Database Metadata Fields

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Documents additional critical metadata fields within the Linear Sync Engine's database. This includes `firstSyncId` for incremental sync starting points, `backendDatabaseVersion` for compatibility tracking, and `updatedAt` for last modification timestamps.

```APIDOC
/**
 * @property {number} firstSyncId - The lastSyncId value at the time of full bootstrapping, used as a starting point for incremental synchronization.
 * @property {string} backendDatabaseVersion - Indicates the version of the backend database for compatibility tracking.
 * @property {string} updatedAt - A timestamp (ISO 8601 string) indicating the last time the database or its metadata was updated.
 */
interface DatabaseMetadata {
  firstSyncId: number;
  backendDatabaseVersion: string;
  updatedAt: string;
}
```

--------------------------------

### GraphQL Request Body for Linear Sync Engine Batch Sync

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This JSON object represents the request body sent to the Linear API's `/sync/batch` endpoint. It includes a `firstSyncId` to indicate the starting point for sync and an array of `requests`, each specifying an `indexedKey`, `keyValue`, and `modelName` to fetch specific models in a batched manner.

```JSON
{
  "firstSyncId": 3528373991,
  "requests": [
    {
      "indexedKey": "issueId",
      "keyValue": "bda1a998-91b0-4ceb-8f89-91b7f6608685",
      "modelName": "Comment"
    },
    {
      "indexedKey": "issueId",
      "keyValue": "bda1a998-91b0-4ceb-8f89-91b7f6608685",
      "modelName": "IssueHistory"
    }
  ]
}
```

--------------------------------

### Example JSON Object for Bootstrap Response Metadata

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet displays the metadata JSON object found as the last line of the streamed bootstrap response. It contains crucial information such as the method, last sync ID, subscribed sync groups, and counts of returned models.

```JSON
{
  "method": "mongo",
  "lastSyncId": 2326713666,
  "subscribedSyncGroups": [
    "89388c30-9823-4b14-8140-4e0650fbb9eb",
    "4e8622c7-0a24-412d-bf38-156e073ab384",
    "AD619ACC-AAAA-4D84-AD23-61DDCA8319A0",
    "CDA201A7-AAAA-45C5-888B-3CE8B747D26B"
  ],
  "databaseVersion": 948,
  "returnedModelsCount": {
    "Activity": 6,
    "Cycle": 2,
    "DocumentContent": 5,
    "Favorite": 1,
    "GitAutomationState": 3,
    "Integration": 1,
    "Issue": 3,
    "IssueLabel": 4,
    "NotificationSubscription": 2,
    "Organization": 1,
    "Project": 2,
    "ProjectStatus": 5,
    "Team": 1,
    "TeamKey": 1,
    "TeamMembership": 1,
    "User": 1,
    "UserSettings": 1,
    "WorkflowState": 7,
    "Initiative": 1,
    "SyncAction": 0
  }
}
```

--------------------------------

### Applying MobX Decorators for Observable Properties in LSE

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet illustrates how the Linear Sync Engine (LSE) utilizes MobX's @Action and @Computed decorators to make model properties observable. These decorators are crucial for tracking changes and enabling features like transaction generation and efficient data synchronization. The first example shows the compiled JavaScript/JSX output, while the second provides the original TypeScript source.

```jsx
Pe([rt], re.prototype, "moveToTeam", null);
Pe([O], re.prototype, "parents", null);
```

```typescript
@ClientModel("Issue")
class Issue {
  @Action
  moveToTeam() {
    // implementation
  }

  @Computed
  get parents() {
    // implementation
  }
}
```

--------------------------------

### Reference Sync Action Types (APIDOC)

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This section defines the various single-character action types found within sync delta packets. Each type signifies a specific operation performed on a model, such as insertion, update, archiving, or deletion, guiding how the client applies changes.

```APIDOC
Possible Sync Action Types:
1. I - Insertion
2. U - Update
3. A - Archiving
4. D - Deletion
5. C - Covering
6. G - Changing sync groups
7. S - Changing sync groups (distinction from G unclear)
8. V - Unarchiving
```

--------------------------------

### Define Model Properties with TypeScript's @Property Decorator

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet demonstrates how to declare model properties using the `@Property` decorator in TypeScript. It shows examples for a simple string property (`title`) and a property with a custom serializer (`priority`), highlighting how these properties are registered and made observable within the Reverse Linear Sync Engine's `ModelRegistry`.

```TypeScript
Pe([w()], re.prototype, "title", void 0);
Pe(
  [
    w({
      serializer: P_,
    }),
  ],
  re.prototype,
  "priority",
  void 0
);
```

```TypeScript
@ClientModel("Issue")
class Issue extends Model {
  @Property()
  public title: string;

  @Property({ serializer: PrioritySerializer })
  public priority: Priority;
}
```

--------------------------------

### LSE: `SyncClient.initializeDatabase` Bootstrapping Process

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Outlines the multi-step process initiated by `SyncClient.initializeDatabase` (or `ng.initializeDatabase`) for full database bootstrapping in IndexedDB. It covers retrieving workspace metadata, creating the workspace-specific database, and verifying store readiness before data loading.

```APIDOC
/**
 * @method initializeDatabase
 * @memberof SyncClient
 * @description Initiates the full bootstrapping process for a Linear Sync Engine database in IndexedDB.
 * @returns {Promise<void>}
 * @steps
 *   1. Retrieve Workspace Metadata:
 *      - Calls `Xn.databaseInfo` to get workspace metadata.
 *      - Initializes `linear_databases` if not present.
 *      - Stores metadata using `Xn.registerDatabase`.
 *   2. Create the Workspace-Specific Database:
 *      - Establishes connection to the workspace database (e.g., `linear_b4782b3125a816b51a44e59f2e939efa`).
 *      - Evaluates if creation or migration is needed.
 *      - Invokes `this.storeManager.createStores(i, l)` to initialize model tables.
 *      - Initializes database metadata fields to defaults if no metadata is stored yet.
 *   3. Check Store Readiness:
 *      - Verifies the readiness of each store (table).
 *      - During first load, all stores are initially empty and not ready.
 */
class SyncClient {
  initializeDatabase(): Promise<void>;
}
```

--------------------------------

### APIDOC: Database.requiredBootstrap Method Details

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The `Database.requiredBootstrap` method identifies the appropriate bootstrapping type and supplies the necessary parameters for its execution. It returns an object containing the determined bootstrapping type, a list of models to load, and the current `lastSyncId`.

```APIDOC
class Database {
  /**
   * Identifies the appropriate bootstrapping type and supplies necessary parameters for its execution.
   * @returns {object} An object describing the required bootstrap operation.
   * @returns {string} return.type - The type of bootstrapping to perform. Possible values:
   *   - 'full': LSE retrieves all required models from the server.
   *   - 'local': Data is loaded from the local database, and the application synchronizes with the server using incremental deltas.
   *   - 'partial': A subset of models is loaded from the server, depending on the load strategy.
   * @returns {string[]} return.modelsToLoad - Names of the models with a load strategy marked as either instant or lazy.
   * @returns {number} return.lastSyncId - Indicates the database snapshot the client is currently synchronized to. During full bootstrapping, this value is zero, as no data has yet been loaded from the server. In this case, the server will returns the latest snapshot.
   */
  static requiredBootstrap(): { type: 'full' | 'local' | 'partial'; modelsToLoad: string[]; lastSyncId: number; };
}
```

--------------------------------

### APIDOC: Linear Sync Engine Model Hydration and Object Pool Process

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This conceptual snippet outlines the steps involved in hydrating raw models into in-memory objects and adding them to the Object Pool. It details the sequence of method calls, including loading initial data, instantiating model objects using constructors from the ModelRegistry, and populating them with data and references before making them accessible via the SyncClient's modelLookup map.

```Conceptual
// Conceptual flow of Model Hydration and Object Pool population
function hydrateModelsAndPopulateObjectPool() {
    // 1. Initiate model bootstrapping
    SyncClient.bootstrap();

    // 2. Load initial hydrated model data from the database
    // (Corresponds to Database.getAllInitialHydratedModelData)
    const initialModels = Database.getAllInitialHydratedModelData();

    // 3. Iterate through models and add them to the Object Pool
    for (const modelData of initialModels) {
        // Retrieve constructor from ModelRegistry
        const ModelConstructor = ModelRegistry.getConstructor(modelData.type);

        // Instantiate model object
        const modelObject = new ModelConstructor();

        // Hydrate the model object with data
        // (Corresponds to ClientModel.updateFromData)
        modelObject.updateFromData(modelData);

        // Resolve and populate references
        // (Corresponds to ClientModel.updateReferencedModels)
        modelObject.updateReferencedModels();

        // Add the hydrated model object to the Object Pool (modelLookup map)
        // (Corresponds to SyncClient.addModelToLiveCollections)
        SyncClient.addModelToLiveCollections(modelObject);
    }
}

// Conceptual Object Pool structure within SyncClient
class SyncClient {
    modelLookup: Map<string, ClientModel>; // Maps model ID to model object

    addModelToLiveCollections(model: ClientModel) {
        this.modelLookup.set(model.id, model);
    }
}
```

--------------------------------

### Understanding Linear Sync Engine Models

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Describes the core concepts of models within the Linear Sync Engine, including their properties, references, loading mechanisms, and how they interact with transactions and the Object Pool.

```APIDOC
Concept: Model
Description: Entities (e.g., Issue, Team, Organization, Comment) with properties and references, many observable via MobX.

Properties:
- Properties: Observable attributes of a model.
- References: Links to other models.
- Metadata: Dictates model behavior.

Loading:
- Source: Local database (IndexedDB) or server.
- Partial Loading: Supports on-demand loading from local DB or server.

Storage:
- Object Pool: A large map storing loaded models, retrievable by UUIDs.

Performance:
- Lazy Hydration: Properties loaded only when accessed to improve performance.

Operations:
- Transactions: Operations (additions, deletions, updates, archiving) encapsulated and sent to the server, then broadcast as delta packets.
```

--------------------------------

### Linear Sync Engine BatchModelLoader API Overview

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This API documentation outlines the core components and methods involved in the `BatchModelLoader` within the Linear Sync Engine. It details how network hydration requests are batched and categorized, and the methods responsible for processing different types of sync requests.

```APIDOC
Class: BatchModelLoader
  Description: Batches multiple network hydration requests into a single GraphQL request.
  Methods:
    handleBatch():
      Description: Categorizes and processes incoming sync requests based on index keys or SyncGroups.
      Internal Calls: loadSyncBatch(), loadPartialModels(), loadFullModels()

    loadSyncBatch():
      Description: Sends a batched sync request to the Linear API's /sync/batch endpoint.
      Endpoint: POST https://client-api.linear.app/sync/batch
      Parameters:
        firstSyncId: number (Required) - The starting sync ID for the batch.
        requests: Array<Object> (Required) - List of model hydration requests.
          - indexedKey: string - The key used for indexing (e.g., "issueId").
          - keyValue: string - The value of the indexed key.
          - modelName: string - The name of the model to hydrate (e.g., "Comment", "IssueHistory").
      Returns: JSON stream of hydrated model objects and metadata.
      Internal Calls: GraphQLClient.resetModelsJsonStream()

    loadPartialModels():
      Description: Handles requests associated with a partial index key, loading corresponding models.

    loadFullModels():
      Description: Handles requests that are neither associated with an indexedKey nor a SyncGroup, loading full models.

Class: GraphQLClient
  Methods:
    resetModelsJsonStream(url: string, body: Object): 
      Description: Initiates a request to the specified URL with the given body, expecting a JSON stream response.

Function: createModelsFromData(data: Object):
  Description: Parses the batch sync response, writes models to the database, creates in-memory objects, and saves partial index information.
  Parameters:
    data: Object - The parsed JSON response from the batch sync endpoint.
```

--------------------------------

### Linear Sync Engine Transaction Operations

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Explains how operations are packaged as transactions, their server-side execution, client-side reversibility, caching, and the role of sync IDs for consistency and undo/redo functionality.

```APIDOC
Concept: Transaction
Description: Operations sent to the server, designed for exclusive server execution and client-side reversibility.

Properties:
- Sync ID: A monotonically increasing number ensuring correct order of operations.

Behavior:
- Server Execution: Intended to execute exclusively on the server.
- Client Reversibility: Can be reversed on the client in case of failure.
- Caching: Temporarily cached in IndexedDB if connection is lost, resent upon reestablishment.
- Consistency: Sync ID maintains consistency across clients.
- Features: Supports undo and redo operations for collaborative workflows.
```

--------------------------------

### GraphQL Response for Batched Project and Document Creation

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet displays the server's response to a batched GraphQL mutation request involving multiple operations. The `data` field contains individual results for each aliased operation (`o1` for project, `o2` for document), each providing its respective `lastSyncId` upon successful completion.

```JSON
{
  "data": {
    "o1": {
      "lastSyncId": 0
    },
    "o2": {
      "lastSyncId": 3588467486
    }
  }
}
```

--------------------------------

### LSE In-Memory Data Update: Model Preparation

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

LSE performs a first loop through sync actions to prepare models for processing. For 'I', 'V', or 'U' actions, corresponding model instances are created. For 'A' actions, existing models' properties are updated. References are then attached, with a check to prevent operations on models deleted within the same delta packet by comparing `syncId`s.

```APIDOC
FUNCTION PrepareModelsForSync(syncActions):
  FOR each action IN syncActions:
    IF action.type IN ['I', 'V', 'U'] THEN
      CreateModelInstance(action.modelData)
    ELSE IF action.type == 'A' THEN
      UpdateModelProperties(action.modelId, action.properties)
  
  FOR each newlyCreatedModel:
    IF NOT IsDeletedByLaterAction(newlyCreatedModel.syncId, deltaPacket.syncActions) THEN
      AttachReferences(newlyCreatedModel)
```

--------------------------------

### LSE Client-Server Transaction Lifecycle

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/SUMMARY.md

This section outlines the six-step process of how transactions are handled in the Local Storage Engine (LSE), from initial property assignment to server processing and cache management. It details the creation, queuing, and batch sending of transactions, along with their caching in IndexedDB for resilience.

```APIDOC
1. Property Assignment:
   - System records: changed property name, previous value.
   - Models in memory: updated immediately.

2. Transaction Creation (e.g., issue.save()):
   - An UpdateTransaction is created, capturing model changes.

3. Transaction Queuing & Caching:
   - UpdateTransaction added to request queue.
   - Saved in __transactions table (IndexedDB) for caching.

4. Batch Sending:
   - TransactionQueue schedules timers to send queued transactions to server in batches.

5. Server Processing & Cache Clearing:
   - Upon successful backend processing, batch removed from __transactions table.
   - LSE clears cached batch.

6. Delta Packet Dependency:
   - Transactions wait for delta packets containing lastSyncId to complete before proceeding.
```

--------------------------------

### LSE Model Loading Strategies (loadStrategy)

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

LSE defines various `loadStrategy` options to control how models are loaded into the client application. These strategies dictate whether models are loaded during bootstrapping, on demand, or only when explicitly requested, optimizing data fetching and application performance. The `local` strategy is for models stored exclusively in the local database.

```APIDOC
enum LoadStrategy {
  instant = "Models that are loaded during application bootstrapping (default strategy).",
  lazy = "Models that do not load during bootstrapping but are fetched all at once when needed (e.g., ExternalUser).",
  partial = "Models that are loaded on demand, meaning only a subset of instances is fetched from the server (e.g., DocumentContent).",
  explicitlyRequested = "Models that are only loaded when explicitly requested (e.g., DocumentContentHistory).",
  local = "Models that are stored exclusively in the local database. No models have been identified using this strategy."
}
```

--------------------------------

### APIDOC: Linear Sync Engine Bootstrapping Response Structure

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet illustrates the structure of the /sync/bootstrap API response. It includes key fields such as 'method' indicating the data source, 'lastSyncId' for client snapshot updates, 'databaseVersion' for schema compatibility, and 'returnedModelsCount' for response validation. Note that 'subscribedSyncGroups' has been moved to a pre-request.

```JSON
{
  "method": "mongo",
  "lastSyncId": "2326713666",
  "databaseVersion": 948,
  "returnedModelsCount": {
    "Activity": 6,
    "Cycle": 2,
    "DocumentContent": 5,
    "Favorite": 1,
    "GitAutomationState": 3,
    "Integration": 1,
    "Issue": 3,
    "IssueLabel": 4,
    "NotificationSubscription": 2,
    "Organization": 1,
    "Project": 2,
    "ProjectStatus": 5,
    "Team": 1,
    "TeamKey": 1,
    "TeamMembership": 1,
    "User": 1,
    "UserSettings": 1,
    "WorkflowState": 7,
    "Initiative": 1,
    "SyncAction": 0
  }
}
```

--------------------------------

### APIDOC: LazyReferenceCollection Hydration Flow

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This section outlines the sequence of method calls involved when `LazyReferenceCollection` initiates a data hydration. It details how the process moves from the collection's `hydrate` method to `SyncClient.hydrateModelsByIndexedKey`, including the key parameters passed for querying models.

```APIDOC
Method: LazyReferenceCollection.hydrate()
  Description: Initiates the hydration process for the collection's models.
  Calls: SyncedStore.hydrateModels

Method: SyncedStore.hydrateModels(modelClass: Class, queryParams: Object)
  Description: Orchestrates model hydration, potentially involving network requests.
  Calls: SyncClient.hydrateModelsByIndexedKey

Method: SyncClient.hydrateModelsByIndexedKey(modelClass: Class, params: Object)
  Description: Fetches models from the server or local database using indexed keys.
  Parameters:
    modelClass: The class of the model to hydrate (e.g., Comment).
    params: An object containing query parameters.
      key: string - Indicates the reference type (e.g., "Issue" for Comment references by Issue).
      coveringPartialIndexValues: Array<string> - Signifies indirect references, used to check if models were previously fetched.
      value: string - The ID of the referencing model (e.g., Issue ID).
  Returns:
    Promise<Array<Model>>: A promise that resolves with the hydrated models.
```

--------------------------------

### TypeScript Handshake Callback Parameters for SyncClient

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This TypeScript snippet illustrates the structure of the parameters passed to the `handshakeCallback` function upon successful WebSocket connection. It includes `userSyncGroups` for different synchronization categories, `lastSyncId` and `lastSequentialSyncId` for tracking synchronization progress, and `databaseVersion` for schema compatibility. These parameters are crucial for determining if any incremental changes were missed during the bootstrapping process.

```typescript
{
  "userSyncGroups": {
    "all": [
      "89388c30-9823-4b14-8140-4e0650fbb9eb",
      "094f76cf-b0c1-4f6c-9908-1801a6654f05",
      "4e8622c7-0a24-412d-bf38-156e073ab384",
      "E7E6104E-AAAA-42BC-9B8B-B91FCDD9946B",
      "AD619ACC-AAAA-4D84-AD23-61DDCA8319A0",
      "CDA201A7-AAAA-45C5-888B-3CE8B747D26B",
      "B0B41C7E-AAAA-4C7D-A93D-CD9565DA4358"
    ],
    "optimized": [
      "4e8622c7-0a24-412d-bf38-156e073ab384",
      "E7E6104E-AAAA-42BC-9B8B-B91FCDD9946B",
      "AD619ACC-AAAA-4D84-AD23-61DDCA8319A0",
      "CDA201A7-AAAA-45C5-888B-3CE8B747D26B",
      "B0B41C7E-AAAA-4C7D-A93D-CD9565DA4358"
    ]
  },
  "lastSyncId": 3529152751,
  "lastSequentialSyncId": 3529152751,
  "databaseVersion": 1179
}
```

--------------------------------

### Defining Models and Metadata with ModelRegistry

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Documents the `ModelRegistry` class, a central component in LSE for managing metadata related to models, their properties, and references. It details key static members used for lookup and registration.

```APIDOC
Class: ModelRegistry
Description: A class with static members that stores various types of metadata for LSE models and provides methods for registering and retrieving this information.

Static Members:
- "modelLookup": Maps a model's name to its constructor.
- "modelPropertyLookup": Stores metadata about a model's properties.
- "modelReferencedPropertyLookup": Stores metadata about a model's references.
- "etc.": Other static members for managing metadata.
```

--------------------------------

### Overview of SyncClient.applyDelta Method Functionality (APIDOC)

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The `ng.applyDelta` method, aliased as `SyncClient.applyDelta`, is central to processing incoming sync actions. It manages user sync group changes, potentially triggering partial bootstrapping, and handles the loading of dependent models required for accurate state updates. This method ensures data consistency across clients.

```APIDOC
Method: SyncClient.applyDelta (aliased as ng.applyDelta)
Purpose: Handles incoming sync actions (delta packets).

Tasks Performed:
1. Determines if the user is added to or removed from sync groups. If added, LSE triggers a network request (partial bootstrapping) to fetch associated models and waits for the response before processing further sync actions.
2. Loads dependencies of specific actions, leveraging `DependentsLoader.supportedPacket`.
```

--------------------------------

### APIDOC: Key Classes and Methods in LSE Hydration

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This section provides a reference to the core classes and their methods involved in the LSE's data hydration and storage mechanisms. These components work together to manage model fetching, local storage, and batch processing of network requests.

```APIDOC
Class: SyncClient (ng)
  Methods:
    hydrateModelsByIndexedKey(modelClass: Class, params: Object): Promise<Array<Model>>
      Description: Hydrates models from the server or local cache based on indexed keys.

Class: Database (eg)
  Methods:
    getModelDataByIndexedKey(modelClass: Class, key: string, value: string): Promise<Array<Model>>
      Description: Retrieves model data from the local database using an indexed key.
    getAllFromIndex(modelClass: Class, indexName: string, query: IDBKeyRange): Promise<Array<Model>>
      Description: Retrieves all models from a specific IndexedDB index.
    setPartialIndexValueForModel(modelClass: Class, partialIndexValue: string, modelId: string): Promise<void>
      Description: Updates the partial index store for a given model.

Class: PartialStore (Jm)
  Methods:
    getAllForIndexedKey(modelClass: Class, key: string): Array<string>
      Description: Retrieves all partial index values for a given model and key.
    hasModelsForPartialIndexValues(modelClass: Class, partialIndexValues: Array<string>): boolean
      Description: Checks if the store contains models for the specified partial index values.

Class: BatchedRequest (PE)
  Methods:
    addRequest(request: Object): void
      Description: Adds a request to the batch for processing.

Class: BatchModelLoader (wm)
  Methods:
    addRequest(request: Object): void
      Description: Adds a model loading request to the batch.
    handleBatch(): Promise<void>
      Description: Processes the accumulated batch of model loading requests.
    loadSyncBatch(requests: Array<Object>): Promise<Array<Model>>
      Description: Loads a batch of models synchronously.
    loadFullModels(modelClass: Class, ids: Array<string>): Promise<Array<Model>>
      Description: Loads full model data for a given set of IDs.
    handleLoadedModels(models: Array<Model>): void
      Description: Processes models after they have been loaded.
```

--------------------------------

### APIDOC: PartialIndexHelper (ku) Class for Model Referencing

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The `PartialIndexHelper` class, internally referred to as `ku`, is central to LSE's ability to determine how models can be referenced by other models. It calculates both direct and indirect (nested) reference paths, supporting up to three levels of depth for complex relationships.

```APIDOC
Class: PartialIndexHelper (ku)
Description: Manages how models can be referenced by other models, including direct and indirect (nested) references.

Methods:
  processPartialIndexInfoForModel(modelClass: Class): Array<PartialIndexInfo>
    Description: Calculates and returns an array describing how a model can be referenced, supporting up to three levels of depth for nested references.
    Parameters:
      modelClass: The class of the model for which to process partial index information.
    Returns:
      Array<PartialIndexInfo>: An array of objects describing reference paths.
```

--------------------------------

### LSE Transient Partial Indexed Key Generation

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The Linear Sync Engine (LSE) initiates synchronization by retrieving transient partial indexed keys for a model. These keys are derived from the Cartesian product of the model's partial indexes and its dependencies, crucial for identifying relevant data subsets.

```APIDOC
FUNCTION GetTransientPartialIndexedKeys(model):
  partialIndexes = model.getPartialIndexes()
  dependencies = model.getDependencies()
  RETURN CartesianProduct(partialIndexes, dependencies)
```

--------------------------------

### Logic for DependentsLoader.supportedPacket (APIDOC)

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

The `DependentsLoader.supportedPacket` method determines which sync actions necessitate loading dependent models, such as child issues for an updated parent issue. This ensures that all related data is available for processing. The method applies specific conditions based on action type, model type, and partial index keys to optimize dependency loading.

```APIDOC
Method: DependentsLoader.supportedPacket (aliased as Zu.supportedPacket)
Purpose: Identifies sync actions that require loading of dependent models.

Conditions for Loading Dependent Models:
1. The sync action type must NOT be "I" (Insertion), "A" (Archiving), or "D" (Deletion).
2. The model manipulated by the action must be either an `Issue` or a `Project`, and it must be used for partial indexes.
3. The action type should be "V" (Unarchiving), OR its associated model should have changed references for which LSE already stores the partial index keys.
```

--------------------------------

### APIDOC: Model and Property Metadata Definitions

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/SUMMARY.md

Defines the structure and types for model and property metadata managed by the `ModelRegistry` in the Linear Sync Engine. This metadata includes loading strategies, partial load modes, and various property types with their attributes.

```APIDOC
{
  "ModelMetadata": {
    "loadStrategy": {
      "type": "enum",
      "values": ["instant", "lazy", "partial", "explicitlyRequested", "local"],
      "description": "Defines how models are loaded into the client."
    },
    "partialLoadMode": {
      "type": "enum",
      "values": ["full", "regular", "lowPriority"],
      "description": "Specifies how a model is hydrated."
    },
    "usedForPartialIndexes": {
      "type": "boolean",
      "description": "Relates to the functionality of partial indexing."
    }
  },
  "PropertyMetadata": {
    "type": {
      "type": "enum",
      "values": ["property", "ephemeralProperty", "reference", "referenceModel", "referenceCollection", "backReference", "referenceArray"],
      "description": "Specifies the property's type and its relationship to the model."
    },
    "lazy": {
      "type": "boolean",
      "description": "Specifies whether the property should be loaded only when the model is hydrated."
    },
    "serializer": {
      "type": "string",
      "description": "Defines how to serialize the property for data transfer or storage."
    },
    "indexed": {
      "type": "boolean",
      "description": "Determines whether the property should be indexed in the database. Used for references."
    },
    "nullable": {
      "type": "boolean",
      "description": "Specifies whether the property can be null, used for references."
    }
  },
  "ModelRegistry": {
    "__schemaHash": {
      "type": "string",
      "description": "A hash of all models' metadata and their properties' metadata, crucial for determining local database migration."
    }
  }
}
```

--------------------------------

### LSE: Understanding `lastSyncId` for Client Synchronization

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Explains the role of `lastSyncId` in determining client synchronization status with the server. It details how a smaller client `lastSyncId` indicates missing delta packets and how this ID is updated during full bootstrapping and subsequent delta packet receptions.

```APIDOC
/**
 * @property {number} lastSyncId - A unique identifier representing the client's synchronized state with the server.
 * @description Used to determine if the client is missing transactions. Updated during full bootstrapping and with each received delta packet.
 * If client's lastSyncId < server's lastSyncId, client is out of sync.
 */
interface DatabaseMetadata {
  lastSyncId: number;
}
```

--------------------------------

### TypeScript: Define LazyReferenceCollection for Assigned Issues

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

This snippet demonstrates the instantiation of a `LazyReferenceCollection` for `Issue` models, allowing them to be loaded based on an `assigneeId`. It configures the collection to potentially skip network hydration under specific conditions, optimizing data retrieval.

```TypeScript
public assignedIssues = new LazyReferenceCollection(
  Issue,
  this,
  "assigneeId",
  undefined,
  {
    canSkipNetworkHydration: () => this.canSkipNetworkHydration(Issue)
  }
);
```

--------------------------------

### APIDOC: UpdateTransaction Generation Flow

Source: https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/README.md

Describes the sequence of method calls leading to the creation of an "UpdateTransaction". The "ClientModel.save" method initiates the process, which flows through "SyncedStore.save" to "SyncClient.update", and finally to "TransactionQueue.update". This method invokes "changeSnapshot" on the model to capture changes and constructs the "UpdateTransaction" instance.

```APIDOC
interface ClientModel {
  save(): void; // Initiates the save process, calling SyncedStore.save.
}

interface SyncedStore {
  save(): void; // Calls SyncClient.update.
}

interface SyncClient {
  update(): void; // Calls TransactionQueue.update.
}

interface TransactionQueue {
  update(): void; // Generates an UpdateTransaction instance.
                          // Invokes model.changeSnapshot() to get change data.
  enqueueTransaction(transaction: UpdateTransaction): void; // Adds transaction to createdTransactions queue.
}
```
Tokens:
Raw