### Start Backend/Host Frontend Source: https://github.com/afedulov/fraud-detection-demo/blob/master/webapp/README.md Compile and run the Spring Boot application using Maven. This command cleans the project, installs dependencies, and starts the application. ```bash mvn clean install spring-boot:run ``` -------------------------------- ### Install React App Dependencies Source: https://github.com/afedulov/fraud-detection-demo/blob/master/webapp/README.md Before starting the React development server, install all necessary project dependencies using npm. ```bash npm install ``` -------------------------------- ### Build and Run Fraud Detection Demo Source: https://github.com/afedulov/fraud-detection-demo/blob/master/README.md Commands to clone the repository, build Docker images for the web application and Flink job, and start the demo services using Docker Compose. Ensure Docker and Docker Compose are installed and have sufficient resources allocated. ```bash git clone https://github.com/afedulov/fraud-detection-demo cd fraud-detection-demo docker build -t demo-fraud-webapp:latest -f webapp/webapp.Dockerfile webapp/ docker build -t flink-job-fraud-demo:latest -f flink-job/Dockerfile flink-job/ docker-compose -f docker-compose-local-job.yaml up ``` -------------------------------- ### Start React App Source: https://github.com/afedulov/fraud-detection-demo/blob/master/webapp/README.md Start the React development server to run the frontend application. This command also implicitly starts the Java app. ```bash npm start ``` -------------------------------- ### Start Kafka Source: https://github.com/afedulov/fraud-detection-demo/blob/master/webapp/README.md Navigate to the demo-backend directory and use docker-compose to start Kafka. The --log-level CRITICAL flag suppresses non-essential output. ```bash cd demo-backend docker-compose --log-level CRITICAL up ``` -------------------------------- ### Start Transaction Generation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Starts the demo transaction generator, which begins publishing transactions to Kafka. ```APIDOC ## GET /api/startTransactionsGeneration ### Description Starts the demo transaction generator, publishing transactions to Kafka. ### Method GET ### Endpoint /api/startTransactionsGeneration ### Response #### Success Response (200 OK or 204 No Content) ``` -------------------------------- ### Typical Session Workflow Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Demonstrates a typical session workflow for the data generation service, from starting slow to stress testing and stopping generation. ```bash # 1. Start slow to observe behavior curl http://localhost:8080/api/generatorSpeed/5 ``` ```bash # 2. Create a rule (if not already created) curl -X POST http://localhost:8080/api/rules \ -H "Content-Type: application/json" \ -d '{"rulePayload":"{...}"}' ``` ```bash # 4. Increase speed for stress testing curl http://localhost:8080/api/generatorSpeed/500 ``` ```bash # 7. Stop generation when done curl http://localhost:8080/api/generatorSpeed/0 ``` -------------------------------- ### Example Rule Input Format Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Provides examples of how to format rules for the Flink job when submitting data via netcat. Rules include state, aggregation keys, unique keys, aggregation fields, functions, limit operators, limits, and window sizes. ```text 1,(active),(paymentType),,(paymentAmount),(SUM),(>),(50),(20) ``` ```text 1,(delete),(paymentType),,(paymentAmount),(SUM),(>),(50),(20) ``` ```text 2,(active),(payeeId),,(paymentAmount),(SUM),(>),(10),(20) ``` ```text 2,(pause),(payeeId),,(paymentAmount),(SUM),(>),(10),(20) ``` -------------------------------- ### Example CLI Parameters Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Demonstrates common command-line interface parameters used to configure the Flink job's data sources, rule sources, and sinks. ```text --data-source kafka --rules-source kafka --alerts-sink kafka --rules-export-sink kafka ``` -------------------------------- ### Start Transaction Generation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Starts the demo transaction generator, which publishes transactions to Kafka. This action executes the `DemoTransactionsGenerator` in a background thread pool and sets an internal flag indicating that transactions are being generated. ```http GET /api/startTransactionsGeneration ``` -------------------------------- ### Start Transactions Generation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Initiates the generation of mock transactions for testing fraud detection scenarios. ```APIDOC ## POST /api/startTransactionsGeneration ### Description Starts the generation of mock transactions. ### Method POST ### Endpoint /api/startTransactionsGeneration ``` -------------------------------- ### Example Rule with Special Function Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Shows an example of a rule definition that utilizes a special aggregation function, COUNT_FLINK, in conjunction with other parameters. ```text 1,(active),(paymentType),,(COUNT_FLINK),(SUM),(>),(50),(20) ``` -------------------------------- ### Start and Control Transaction Generation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Initiates the generation of mock transactions and allows setting the speed of generation. Use this to simulate traffic for testing rule triggers. ```bash # Start generating transactions curl http://localhost:5656/api/startTransactionsGeneration ``` ```bash # Set speed to 10 transactions/second curl http://localhost:5656/api/generatorSpeed/10 ``` -------------------------------- ### Example Rule JSON Source: https://github.com/afedulov/fraud-detection-demo/blob/master/webapp/README.md This is an example of a JSON object representing a rule configuration for the fraud detection system. It includes parameters for rule ID, state, grouping, aggregation, and limits. ```json { "ruleId":1, "ruleState":"ACTIVE", "groupingKeyNames":[ "paymentType" ], "unique":[], "aggregateFieldName":"paymentAmount", "aggregatorFunctionType":"SUM", "limitOperatorType":"GREATER", "limit":50, "windowMinutes":20 } ``` -------------------------------- ### Transaction Model JSON Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Illustrates the JSON format for a financial transaction, including all relevant fields. ```json { "transactionId": 5954524216210268000, "eventTime": 1579612200000, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 } ``` -------------------------------- ### Flink Rule Method: getWindowStartFor Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/types.md Calculates the start of the aggregation window for a given event timestamp. ```java public long getWindowStartFor(Long timestamp) // Calculates the start of the aggregation window for a given timestamp. // Parameters: // - `timestamp` (Long) — Event timestamp in milliseconds // Returns: Long — Window start timestamp ``` -------------------------------- ### Flink Keyed Usage Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/types.md Demonstrates how to use the Keyed wrapper within a Flink pipeline, showing data stream processing and key-based partitioning for subsequent operations. ```java // After DynamicKeyFunction processing DataStream> keyed = ... .connect(rulesStream) .process(new DynamicKeyFunction()); // Then keyed by the extracted key keyed.keyBy((keyed) -> keyed.getKey()) .connect(rulesStream) .process(new DynamicAlertFunction()); ``` -------------------------------- ### Java Keyed Generic Model Instance Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Illustrates an instance of the Keyed generic model, showing how a Transaction object is wrapped with a String key and an Integer ID. ```java Keyed { wrapped: Transaction { transactionId: 5954524..., payeeId: 20908, ... }, key: "20908", // Extracted from groupingKeyNames id: 1 // Rule ID } ``` -------------------------------- ### Invalid CSV Rule Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md An example of an invalid CSV rule line with only 8 tokens, demonstrating a common cause for IOException due to incorrect field count. ```text 1,ACTIVE,(payeeId),(),paymentAmount,SUM,>,1000 ``` -------------------------------- ### Start Netcat Listener Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Use netcat to listen on a specified port for incoming data. This is used to simulate a data source for the Flink job. ```bash nc -lk 9999 ``` -------------------------------- ### Retrieve All Rules Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RuleRestController.md GET endpoint to fetch all rules from the database. Returns a list of all stored rules. ```java @GetMapping("/rules") List all() ``` ```json [ { "id": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}" }, { "id": 2, "rulePayload": "{\"ruleId\":2, ...}" } ] ``` ```http GET http://localhost:8080/api/rules ``` ```bash curl http://localhost:8080/api/rules ``` ```javascript fetch('http://localhost:8080/api/rules') .then(r => r.json()) .then(rules => console.log(rules)); ``` -------------------------------- ### Alert Model JSON Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Illustrates the JSON format for a fraud alert, showing nested transaction details. ```json { "ruleId": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",...}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 1050.00 } ``` -------------------------------- ### Java Method Signature Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/MANIFEST.md Illustrates a typical Java method signature using Flink/Spring syntax, including annotations, path variables, and exception handling. ```java @GetMapping("/rules/{id}") Rule one(@PathVariable Integer id) throws RuleNotFoundException ``` -------------------------------- ### Example Execution of mockAlert Endpoint Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Illustrates the sequence of operations when the mockAlert endpoint is called, including repository access, transaction retrieval, alert creation, serialization, and WebSocket broadcast. ```text GET http://localhost:8080/api/rules/1/alert ├─ repository.findById(1) │ └─ Rule { id: 1, rulePayload: "..." } ├─ transactionsPusher.getLastTransaction() │ └─ Transaction { paymentAmount: 13.54, ... } ├─ Alert { ruleId: 1, triggeringValue: 135.40 } ├─ mapper.writeValueAsString(alert) │ └─ "{\"ruleId\":1,...,\"triggeringValue\":135.40}" ├─ simpSender.convertAndSend("/topic/alerts", json) │ └─ WebSocket clients receive the alert └─ HTTP 200 OK with alert JSON in body ``` -------------------------------- ### Start Transaction Generation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Initiates the generation of transactions. This is a prerequisite for certain operations, such as generating alerts that depend on transaction data, to prevent NullPointerExceptions. ```APIDOC ## GET /api/startTransactionsGeneration ### Description Starts the process of generating transactions within the system. This endpoint should be called before attempting to generate alerts that rely on transaction data to avoid potential `NullPointerException` errors. ### Method GET ### Endpoint /api/startTransactionsGeneration ### Response #### Success Response (200) - Indicates that transaction generation has been successfully initiated. ### Request Example ```bash curl http://localhost:8080/api/startTransactionsGeneration ``` ``` -------------------------------- ### Start Transactions Generation API Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Starts the transaction generator in a background thread. Transactions are published to Kafka continuously at the configured rate. Idempotent if the generator is already running. ```java @GetMapping("/api/startTransactionsGeneration") public void startTransactionsGeneration() throws Exception ``` ```bash curl http://localhost:8080/api/startTransactionsGeneration ``` -------------------------------- ### Example: SUM Rule Evaluation Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DynamicAlertFunction.md Demonstrates the execution flow for a rule that sums payment amounts within a 10-minute window per payeeId. It shows the state updates, aggregation, and rule evaluation process, including when an alert is emitted. ```text Transaction: payeeId=20908, paymentAmount=150.00 ├─ Add to windowState[1234567890000] = {tx1} ├─ Retrieve Rule from broadcast ├─ Window: [1234567890000 - 600000, 1234567890000] ├─ Aggregate: Set of transactions → sum = 150.00 ├─ Evaluate: 150.00 > 1000? → false └─ No alert emitted (More transactions arrive...) Transaction: payeeId=20908, paymentAmount=900.00 ├─ Aggregate: sum = 1050.00 ├─ Evaluate: 1050.00 > 1000? → true └─ Emit Alert with triggeringValue=1050.00, triggeringEvent=tx2 ``` -------------------------------- ### NumberFormatException Stack Trace Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Shows a typical stack trace when a NumberFormatException occurs during string parsing, indicating an invalid numeric input. ```text com.ververica.field.dynamicrules.Transaction.fromString() Caused by: java.lang.NumberFormatException: For input string: "invalid" ``` -------------------------------- ### Example JSON Rule Format Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Illustrates the JSON format for defining rules, including rule ID, state, grouping keys, aggregation details, limit, and window size. ```json { "ruleId": 1, "ruleState": "ACTIVE", "groupingKeyNames": ["paymentType"], "unique": [], "aggregateFieldName": "paymentAmount", "aggregatorFunctionType": "SUM","limitOperatorType": "GREATER","limit": 500, "windowMinutes": 20} ``` -------------------------------- ### startTransactionsGeneration Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Starts the transaction generator in a background thread. Transactions are published to Kafka continuously at the configured rate. Idempotent if already running. ```APIDOC ## GET /api/startTransactionsGeneration ### Description Starts the transaction generator in a background thread, publishing transactions to Kafka continuously at the configured rate. This operation is idempotent if the generator is already running. ### Method GET ### Endpoint /api/startTransactionsGeneration ### Parameters None ### Request Example ```bash curl http://localhost:8080/api/startTransactionsGeneration ``` ### Response #### Success Response (200 or 204) void #### Response Example None ``` -------------------------------- ### Flink Job Local Execution with Kafka Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/configuration.md Example of running the Flink job locally with Kafka as the data and rules source. This configuration enables checkpointing and specifies a checkpoint interval. ```bash java -cp flink-job.jar com.ververica.field.dynamicrules.Main \ --kafka-host kafka \ --kafka-port 9092 \ --data-source KAFKA \ --rules-source KAFKA \ --local true \ --checkpoints true \ --checkpoint-interval 30000 ``` -------------------------------- ### RulesEvaluator Main Entry Point Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md This Java snippet shows the typical entry point for the RulesEvaluator application. It demonstrates how to parse command-line arguments, create a Config object, instantiate RulesEvaluator, and start the evaluation process. ```java public static void main(String[] args) throws Exception { ParameterTool tool = ParameterTool.fromArgs(args); Parameters inputParams = new Parameters(tool); Config config = new Config(inputParams, STRING_PARAMS, INT_PARAMS, BOOL_PARAMS); RulesEvaluator rulesEvaluator = new RulesEvaluator(config); rulesEvaluator.run(); // Blocks until job is stopped } ``` -------------------------------- ### Retrieve All Rules Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RuleRestController.md Fetches a list of all rules currently stored in the database. This endpoint is useful for getting an overview of all active rules. ```APIDOC ## GET /api/rules ### Description Retrieves all rules from the database. ### Method GET ### Endpoint /api/rules ### Response #### Success Response (200) - **List** — All stored rules ### Response Example ```json [ { "id": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}" }, { "id": 2, "rulePayload": "{\"ruleId\":2,...}" } ] ``` ``` -------------------------------- ### Transaction Model fromString() Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Parses a CSV string into a Transaction object. Expects the format: transactionId,eventTime,payeeId,beneficiaryId,paymentType,paymentAmount. ```text 5954524216210268000,2020-01-15 10:30:00,20908,42694,CRD,13.54 ``` -------------------------------- ### CSV Transaction Format Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Defines the structure for transaction data in CSV format, including transaction ID, event time, and payment details. ```csv transactionId,eventTime,payeeId,beneficiaryId,paymentType,paymentAmount 5954524216210268000,2020-01-15 10:30:00,20908,42694,CRD,13.54 ``` -------------------------------- ### Mock KafkaTemplate for Unit Testing Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/FlinkRulesService.md Demonstrates how to mock KafkaTemplate in unit tests to verify interactions with the FlinkRulesService. This setup allows testing the service logic without a real Kafka broker. ```java @MockBean private KafkaTemplate kafkaTemplate; @Test void testAddRule() { Mockito.when(kafkaTemplate.send(any(), any())).thenReturn(mock(ListenableFuture.class)); flinkRulesService.addRule(testRule); verify(kafkaTemplate).send(eq("rules"), eq(expectedPayload)); } ``` -------------------------------- ### JSON Rule Entity Format Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Shows the structure for a rule entity in JSON format, containing a rule ID and the rule payload as a nested JSON string. ```json { "id": 1, "rulePayload": "{...}" // Nested JSON as string } ``` -------------------------------- ### JSON Rule Payload Format Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Specifies the structure for a rule payload in JSON format, including rule ID, state, aggregation details, and limit conditions. ```json { "ruleId": 1, "ruleState": "ACTIVE", "groupingKeyNames": ["payeeId"], "unique": [], "aggregateFieldName": "paymentAmount", "aggregatorFunctionType": "SUM", "limitOperatorType": ">", "limit": "1000.00", "windowMinutes": 10, "controlType": null } ``` -------------------------------- ### Set Flink Logging Level Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md This command demonstrates how to set Flink's logging level, typically done via environment variables or configuration files before starting a Flink job. ```bash # Set Flink logging level export FLINK_LOG_DIR=/tmp/flink-logs java -cp flink-job.jar -Dlog4j.configuration=file:conf/log4j.properties ... ``` -------------------------------- ### Example of Invalid Transaction Line Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Provides an example of a malformed transaction line that would cause a NumberFormatException during parsing in Flink. ```text 5954524216210268000,2020-01-15 10:30:00,20908,42694,CRD,invalid_amount,1234567890 ``` -------------------------------- ### List Kafka Topics Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Use this command to list all available topics and their partitions in Kafka. Ensure Kafka is running on localhost:9092. ```bash # List topics and partitions kafka-topics.sh --bootstrap-server localhost:9092 --list ``` -------------------------------- ### Mock Alert Response Body Example Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Example JSON structure of the alert object returned by the mockAlert endpoint. This represents a synthetic alert triggered by a transaction. ```json { "ruleId": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",...}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 135.40 } ``` -------------------------------- ### open() Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DynamicAlertFunction.md Initializes the DynamicAlertFunction on a task manager instance. It sets up internal state and registers metrics for monitoring. ```APIDOC ## open() ### Description Called once per task manager instance when the function is initialized. Sets up state and metrics. ### Method Signature ```java public void open(Configuration parameters) ``` ### Side Effects 1. Initializes `windowState` MapState from runtime context. 2. Creates and registers `alertMeter` metric (gauge per second). 3. Metric is visible in Flink web UI under task metrics. ### Flink Runtime Called before any `processElement()` or `processBroadcastElement()` invocations. ``` -------------------------------- ### Activate Spring Profiles Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/configuration.md Use command-line arguments to activate specific Spring Profiles for different environments like local or cloud. ```bash java -jar webapp.jar --spring.profiles.active=local ``` ```bash java -jar webapp.jar --spring.profiles.active=cloud ``` -------------------------------- ### run() Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md Builds and executes the Flink streaming job. This is the main entry point after instantiation. ```APIDOC ## run() ### Description Builds and executes the Flink streaming job. This is the main entry point after instantiation. It sets up the execution environment, checkpoint configuration, data streams, processing pipeline, output sinks, and metrics/logging. ### Method Signature ```java public void run() throws Exception ``` ### Throws - **Exception** — Any exception from Flink environment setup or job execution (TimeoutException, IOException, etc.) ### Side Effects - Creates StreamExecutionEnvironment - Configures Flink checkpointing if enabled - Adds sources for transactions and rules from Kafka/GCP/Generator - Adds sinks for alerts, latency metrics, and rule exports - Calls `env.execute("Fraud Detection Engine")` — blocks until job completes or is cancelled ``` -------------------------------- ### Get Rule by ID Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Retrieves a single rule by its ID. Throws RuleNotFoundException if the rule does not exist. ```http GET /api/rules/{id} ``` ```json { "id": 1, "rulePayload": "{...}" } ``` -------------------------------- ### Project File Organization Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/INDEX.md Illustrates the directory structure of the Fraud Detection Demo project, outlining the purpose of key files and subdirectories. ```text output/ ├── README.md # Project overview ├── types.md # Type definitions ├── endpoints.md # HTTP API specification ├── configuration.md # All configuration options ├── errors.md # Exception handling ├── INDEX.md # This file └── api-reference/ ├── RuleRestController.md # Rule management REST API ├── AlertsController.md # Alert generation and WebSocket ├── DataGenerationController.md # Transaction generation control ├── FlinkRulesService.md # Kafka rule publishing ├── RulesEvaluator.md # Main Flink job ├── DynamicAlertFunction.md # Rule evaluation logic └── Models.md # All data models ``` -------------------------------- ### Get Rule by ID Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Retrieves a single fraud detection rule by its unique identifier. Throws `RuleNotFoundException` if the rule does not exist. ```APIDOC ## GET /api/rules/{id} ### Description Retrieves a single rule by ID. ### Method GET ### Endpoint /api/rules/{id} ### Parameters #### Path Parameters - **id** (Integer) - Required - Rule ID ### Response #### Success Response (200 OK) - **id** (integer) - The unique identifier for the rule. - **rulePayload** (string) - The JSON-serialized rule payload. ### Response Example ```json { "id": 1, "rulePayload": "{...}" } ``` ``` -------------------------------- ### Create a Rule and Generate Mock Alert Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Use these curl commands to create a rule and then generate a mock alert for it. This is useful for testing alert display and integration. ```bash # Create a rule curl -X POST http://localhost:8080/api/rules \ -H "Content-Type: application/json" \ -d '{"rulePayload":"{\"ruleState\":\"ACTIVE\",...}"}' # Generate mock alert for rule 1 curl http://localhost:8080/api/rules/1/alert ``` -------------------------------- ### Retrieve Single Rule by ID Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RuleRestController.md GET endpoint to fetch a specific rule by its database ID. Throws RuleNotFoundException if the ID does not exist. ```java @GetMapping("/rules/{id}") Rule one(@PathVariable Integer id) ``` ```bash curl http://localhost:8080/api/rules/1 ``` ```json # Response: # { # "id": 1, # "rulePayload": "{...}" # } ``` -------------------------------- ### Rule JSON Representation (REST API) Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Example JSON response for a Rule entity from the REST API. Note that rulePayload is a JSON string. ```json { "id": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}" } ``` -------------------------------- ### Project Document Structure Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/MANIFEST.md Illustrates the hierarchical organization of the project's documentation files, showing the relationship between the README, type definitions, API endpoints, configuration, error handling, and detailed API references. ```tree README ├─ Project identity and architecture │ ├─ types.md ├─ Data model definitions │ ├─ endpoints.md ├─ HTTP API contract │ ├─ configuration.md ├─ All tunable parameters │ ├─ errors.md ├─ Exception handling │ └─ api-reference/ ├─ RuleRestController.md ├─ AlertsController.md ├─ DataGenerationController.md ├─ FlinkRulesService.md ├─ RulesEvaluator.md ├─ DynamicAlertFunction.md └─ Models.md ``` -------------------------------- ### Delete Rule Message Payload Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/FlinkRulesService.md This is an example of the JSON payload sent when deleting a rule. Flink primarily checks the ruleId and ruleState fields. ```json { "ruleId": 1, "ruleState": "DELETE" } ``` -------------------------------- ### Project Structure Overview Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/README.md Displays the directory structure of the Fraud Detection Demo project, outlining the organization of the Flink job and web application components. ```text fraud-detection-demo/ ├── flink-job/ │ └── src/main/java/com/ververica/field/ │ ├── dynamicrules/ # Core Flink logic │ ├── config/ # Configuration management │ └── sources/ sinks/ # Data sources and sinks ├── webapp/ │ ├── src/main/java/com/ververica/demo/backend/ │ │ ├── controllers/ # REST API endpoints │ │ ├── services/ # Business logic │ │ ├── entities/ # Domain models │ │ ├── model/ # DTOs and payloads │ │ ├── datasource/ # Data generators │ │ ├── repositories/ # Data access │ │ └── configurations/ # Spring config │ └── src/app/ │ ├── components/ # React components │ ├── interfaces/ # TypeScript types │ └── utils/ # Helper utilities └── docker-compose-local-job.yaml # Full system orchestration ``` -------------------------------- ### Rule Creation Flow Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/FlinkRulesService.md Illustrates the sequence of operations when a new rule is created, involving database saves and broadcasting to Flink via Kafka. ```text RuleRestController.newRule() ├─ repository.save(newRule) // Generate ID ├─ Update payload with assigned ruleId ├─ repository.save(updatedRule) // Save again └─ flinkRulesService.addRule(rule) // Broadcast to Flink └─ kafkaTemplate.send() → Kafka └─ Flink RulesSource receives └─ DynamicAlertFunction.processBroadcastElement() └─ Broadcast state updated ``` -------------------------------- ### SimpleBoundedOutOfOrdernessTimestampExtractor Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md A custom timestamp extractor that gets the event time from Transaction objects and allows a configured watermark delay. Use this when events might arrive out of order. ```java private static class SimpleBoundedOutOfOrdernessTimestampExtractor extends BoundedOutOfOrdernessTimestampExtractor { public SimpleBoundedOutOfOrdernessTimestampExtractor(int outOfOrderdnessMillis); @Override public long extractTimestamp(T element); } ``` -------------------------------- ### Get Rules Source Type Enum Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md Parses the RULES_SOURCE configuration parameter to return the corresponding RulesSource.Type enum. Throws an IllegalArgumentException if the parameter value is invalid. ```java private RulesSource.Type getRulesSourceType() ``` -------------------------------- ### Configure Flink StreamExecutionEnvironment Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md Initializes and configures the Flink StreamExecutionEnvironment with global settings. It applies event time semantics, checkpointing, and sets up the local or cluster environment, including the restart strategy. ```java private StreamExecutionEnvironment configureStreamExecutionEnvironment( RulesSource.Type rulesSourceEnumType, boolean isLocal) ``` -------------------------------- ### Transaction Processing Flow Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/Models.md Illustrates the data flow from a transaction event through keying, aggregation, and alert generation based on rules. ```text Transaction (event) ↓ DynamicKeyFunction ↓ Keyed ↓ keyBy(key) ↓ DynamicAlertFunction (+ Rule from broadcast) ↓ ├─→ if rule.apply(aggregatedValue) │ └─→ Alert(ruleId, rule, key, triggeringEvent, triggeringValue) ``` -------------------------------- ### Usage Example for RuleNotFoundException Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Demonstrates how to use RuleNotFoundException in a Spring REST controller when a rule is not found by its ID. It uses Java's Optional.orElseThrow for concise error handling. ```java // In RuleRestController.one() Rule one(@PathVariable Integer id) { return repository.findById(id) .orElseThrow(() -> new RuleNotFoundException(id)); } ``` -------------------------------- ### RulesEvaluator run() Method Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/RulesEvaluator.md Builds and executes the Flink streaming job. This is the main entry point after instantiation. It sets up the execution environment, checkpoint configuration, data streams, processing pipeline, output sinks, and metrics/logging. ```java public void run() throws Exception ``` -------------------------------- ### Initialize Alert Meter Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DynamicAlertFunction.md Sets up a Flink Meter to track the rate of alerts generated per second. This metric is visible in the Flink Web UI and helps monitor rule violation frequency. ```java private Meter alertMeter; // In open(): alertMeter = new MeterView(60); // 60-second measurement window getRuntimeContext().getMetricGroup().meter("alertsPerSecond", alertMeter); // In processElement(): if (ruleResult) { alertMeter.markEvent(); } ``` -------------------------------- ### Retrieve Kafka Message Listener Container Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Retrieve a Kafka message listener container by its ID from the KafkaListenerEndpointRegistry. This is useful for managing consumer behavior, such as stopping or starting message consumption. ```java MessageListenerContainer listenerContainer = kafkaListenerEndpointRegistry.getListenerContainer(transactionListenerId); ``` -------------------------------- ### Log Mock Alerts Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Logs each mock alert generated. Check application logs for alert creation events. ```java log.info("{}", alert); // Logs each mock alert ``` -------------------------------- ### Add Rule Message Payload Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/FlinkRulesService.md This is an example of the JSON payload sent when adding a rule. It includes details like rule ID, state, grouping keys, and limit conditions. ```json { "ruleId": 1, "ruleState": "ACTIVE", "groupingKeyNames": ["payeeId"], "unique": [], "aggregateFieldName": "paymentAmount", "aggregatorFunctionType": "SUM", "limitOperatorType": ">", "limit": 1000.00, "windowMinutes": 10, "controlType": null } ``` -------------------------------- ### List All Rules Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Retrieves all fraud detection rules from the database. The response contains a list of Rule objects. ```http GET /api/rules ``` ```json [ { "id": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}" } ] ``` -------------------------------- ### Get Alert by Rule ID Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Retrieves details about an alert associated with a specific rule ID. This endpoint can be used to fetch alert information, including the triggering event and value. ```APIDOC ## GET /api/rules/{ruleId}/alert ### Description Retrieves details for a specific alert based on its rule ID. ### Method GET ### Endpoint /api/rules/{ruleId}/alert ### Parameters #### Path Parameters - **ruleId** (integer) - Required - The ID of the rule to retrieve alerts for. ### Response #### Success Response (200) - **ruleId** (integer) - The ID of the rule. - **rulePayload** (string) - The JSON string representing the rule's configuration. - **triggeringEvent** (object) - Details about the event that triggered the alert. - **transactionId** (long) - The ID of the transaction. - **eventTime** (long) - The timestamp of the event. - **payeeId** (integer) - The ID of the payee. - **beneficiaryId** (integer) - The ID of the beneficiary. - **paymentType** (string) - The type of payment. - **paymentAmount** (float) - The amount of the payment. - **triggeringValue** (float) - The calculated value that triggered the alert. ### Response Example ```json { "ruleId": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 135.40 } ``` ``` -------------------------------- ### Enable Backend Debug Logging Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Configure backend logging levels to DEBUG for detailed diagnostics. This is useful for troubleshooting application-specific issues. ```properties logging.level.com.ververica.demo.backend=DEBUG logging.level.org.springframework.kafka=DEBUG ``` -------------------------------- ### Spring Kafka Consumer Error Handling Configuration Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Configuration properties for Spring Kafka to manually acknowledge messages and set a poll timeout. This is part of the consumer error handler setup. ```properties spring.kafka.listener.ack-mode=MANUAL spring.kafka.listener.poll-timeout=3000 ``` -------------------------------- ### Configure WebSocket Alert Topic Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Sets the Kafka topic for WebSocket alerts. This property is configured in application.properties. ```properties web-socket.topic.alerts=/topic/alerts ``` -------------------------------- ### Docker Compose Local Job Environment Variables Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/configuration.md Key environment variables passed to containers in the Docker Compose setup for local development. These configure services like Kafka and Flink. ```yaml environment: KAFKA_HOST: kafka KAFKA_PORT: 9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092 FLINK_JOBMANAGER_RPC_ADDRESS: jobmanager ``` -------------------------------- ### DynamicAlertFunction Methods Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DynamicAlertFunction.md This section details the core methods of the DynamicAlertFunction, including initialization, processing of transaction elements, and handling of broadcast rule updates. ```APIDOC ## DynamicAlertFunction ### Description Implements the core rule evaluation and alert generation logic. A KeyedBroadcastProcessFunction that maintains windowed state per key and evaluates rules dynamically. Handles state cleanup, metrics collection, and control commands. ### Methods #### `open(Configuration parameters)` **Description:** Initializes the function, including setting up state and metrics. **Type:** Overridden method from Flink's ProcessFunction. #### `processElement(Keyed value, ReadOnlyContext ctx, Collector out)` **Description:** Processes incoming transaction elements. It evaluates rules against the transaction data and outputs alerts if rules are violated. **Parameters:** - **value** (Keyed) - Required - The incoming keyed transaction data. - **ctx** (ReadOnlyContext) - Required - The Flink processing context. - **out** (Collector) - Required - Collector for outputting generated alerts. **Type:** Overridden method from KeyedBroadcastProcessFunction. #### `processBroadcastElement(Rule rule, Context ctx, Collector out)` **Description:** Processes incoming broadcast elements, which are rule updates. It updates the internal rule set based on these updates. **Parameters:** - **rule** (Rule) - Required - The incoming rule update. - **ctx** (Context) - Required - The Flink processing context. - **out** (Collector) - Required - Collector for outputting generated alerts (though typically rules don't directly output alerts here, they update state). **Type:** Overridden method from KeyedBroadcastProcessFunction. ### Type Parameters | Parameter | Bound | Meaning | |-----------|-------|---------| | String | Object | Grouping key extracted from transactions | | Keyed | Input | Stream element: transaction with key and rule ID | | Rule | Broadcast | Rule updates from broadcast stream | | Alert | Output | Alert generated when rule is violated | ``` -------------------------------- ### Subscribe to Alerts WebSocket Topic (JavaScript) Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Connects to the WebSocket server and subscribes to the alert topic. Parses incoming JSON messages and logs them. ```javascript import Stomp from 'stompjs'; const stompClient = Stomp.over(new WebSocket('ws://localhost:8080/socket.io')); stompClient.connect({}, () => { stompClient.subscribe('/topic/alerts', (message) => { const alert = JSON.parse(message.body); console.log('Alert received:', alert); }); }); ``` -------------------------------- ### Control Commands for Rule Management Source: https://github.com/afedulov/fraud-detection-demo/blob/master/flink-job/README.md Examples of control commands in JSON format to manage rules and state within the Flink job, such as deleting all rules, exporting current rules, or clearing all state. ```json {"ruleState": "CONTROL", "controlType":"DELETE_RULES_ALL"} ``` ```json {"ruleState": "CONTROL", "controlType":"EXPORT_RULES_CURRENT"} ``` ```json {"ruleState": "CONTROL", "controlType":"CLEAR_STATE_ALL"} ``` -------------------------------- ### Example Invalid RulePayload JSON Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md This JSON structure represents an invalid `rulePayload` that would cause deserialization errors in Jackson due to an unrecognized enum value for `ruleState`. This typically results in an HTTP 500 response. ```json { "rulePayload": "{\"ruleState\":\"INVALID_STATE\",..." } ``` -------------------------------- ### Alert Generation and Streaming Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/SUMMARY.txt Endpoints for simulating alerts and subscribing to real-time fraud alert streams via WebSockets. ```APIDOC ## POST /alerts/mock ### Description Generates a mock alert for testing purposes. ### Method POST ### Endpoint /alerts/mock ### Request Body - alert (object) - The mock alert details. - type (string) - Required - The type of alert. - value (any) - Required - The associated value or data for the alert. ### Request Example { "alert": { "type": "HIGH_RISK_TRANSACTION", "value": { "transactionId": "txn-789", "amount": 5000, "userId": "user-abc" } } } ### Response #### Success Response (200) - message (string) - Confirmation message. #### Response Example { "message": "Mock alert generated." } ## SUBSCRIBE /topic/alerts ### Description Subscribes to a WebSocket stream for real-time fraud alerts. ### Method SUBSCRIBE ### Endpoint /topic/alerts ### Description Clients can subscribe to this endpoint to receive real-time fraud alerts as they are detected by the system. This typically involves establishing a WebSocket connection and subscribing to the relevant message topic. ### Response #### Success Response (WebSocket Message) - alert (object) - The detected fraud alert. - id (string) - Unique identifier for the alert. - timestamp (string) - Time when the alert was generated. - type (string) - Type of fraud detected. - details (object) - Additional details about the alert. #### Response Example { "alert": { "id": "alert-xyz", "timestamp": "2023-10-27T10:30:00Z", "type": "UNUSUAL_SPENDING_PATTERN", "details": { "userId": "user-123", "amount": 1500, "location": "Unknown" } } } ``` -------------------------------- ### React Frontend Proxy Configuration Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/configuration.md The 'proxy' field in package.json is used by the development server to proxy API requests to the backend. This is useful during local development to avoid CORS issues. ```json { "proxy": "http://localhost:8080" } ``` -------------------------------- ### Get Alert by Rule ID Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/AlertsController.md Demonstrates how to retrieve alert details for a specific rule using a cURL request. The response includes the rule configuration, triggering event, and calculated triggering value. Alerts are also sent via WebSocket. ```bash # Request curl -v http://localhost:8080/api/rules/1/alert # Response Headers HTTP/1.1 200 OK Content-Type: application/json # Response Body { "ruleId": 1, "rulePayload": "{\"ruleId\":1,\"ruleState\":\"ACTIVE\",\"groupingKeyNames\":[\"payeeId\"],\"aggregateFieldName\":\"paymentAmount\",\"aggregatorFunctionType\":\"SUM\",\"limitOperatorType\":\">\",\"limit\":1000.00,\"windowMinutes\":10}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 135.40 } # WebSocket Message Sent To /topic/alerts: # (same JSON as above) ``` -------------------------------- ### Consume Kafka Messages from Rules Topic Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/errors.md Use the Kafka console consumer to list all messages from the 'rules' topic. This helps in verifying if rules are being published correctly. ```bash # List messages on rules topic kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic rules --from-beginning ``` -------------------------------- ### Initialize DynamicAlertFunction Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DynamicAlertFunction.md Called once per task manager instance to set up state and metrics before processing elements. Initializes window state, registers an alert metric, and is visible in the Flink web UI. ```java @Override public void open(Configuration parameters) ``` -------------------------------- ### Mock Alert for Rule Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Generates a mock alert for a specific rule, useful for testing purposes. It uses the last received transaction and multiplies the payment amount by 10 to create the triggering value. ```APIDOC ## GET /api/rules/{id}/alert ### Description Generates a mock alert for testing. Uses the last transaction received and creates an alert with a multiplied payment amount (10x) as the triggering value. ### Method GET ### Endpoint /api/rules/{id}/alert #### Path Parameters - **id** (Integer) - Required - Rule ID to trigger alert for ### Response #### Success Response (200 OK) - **ruleId** (Integer) - Description - **rulePayload** (string) - Description - **triggeringEvent** (object) - Description - **transactionId** (long) - Description - **eventTime** (long) - Description - **payeeId** (integer) - Description - **beneficiaryId** (integer) - Description - **paymentType** (string) - Description - **paymentAmount** (double) - Description - **triggeringValue** (double) - Description ### Response Example { "ruleId": 1, "rulePayload": "{...}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 135.40 } ``` -------------------------------- ### Trigger a Mock Alert for a Rule Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Manually triggers a mock alert for a specified rule. This is helpful for testing alert mechanisms without waiting for actual transaction data. ```bash # Trigger mock alert for rule 1 curl http://localhost:5656/api/rules/1/alert ``` -------------------------------- ### Generator State Logs Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/api-reference/DataGenerationController.md Observe generator state changes through log messages. Look for 'startTransactionsGeneration called' and 'Generator speed change request' to track activity. ```log INFO com.ververica.demo.backend.controllers.DataGenerationController - startTransactionsGeneration called INFO com.ververica.demo.backend.controllers.DataGenerationController - Generator speed change request: 100 ``` -------------------------------- ### Mock Alert for Rule Source: https://github.com/afedulov/fraud-detection-demo/blob/master/_autodocs/endpoints.md Generates a mock alert for a specific rule, useful for testing. It uses the last received transaction and multiplies its payment amount by 10 to create the triggering value. The response includes details of the rule, the triggering event, and the calculated triggering value. This endpoint also sends the alert JSON to a WebSocket endpoint and logs the alert. ```http GET /api/rules/{id}/alert ``` ```json { "ruleId": 1, "rulePayload": "{...}", "triggeringEvent": { "transactionId": 5954524216210268000, "eventTime": 1565965071385, "payeeId": 20908, "beneficiaryId": 42694, "paymentType": "CRD", "paymentAmount": 13.54 }, "triggeringValue": 135.40 } ```