### Creating a New Rule: YAML with Options Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md An example of a YAML test case configuration that includes options for verbose and init modes. This is a starting point for defining new rules and testing them. ```yaml - test: # options: # - 'verbose' # - 'init' ``` -------------------------------- ### Example User-Agent String Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/expect/_index.md This is an example of a User-Agent string from a mobile device. ```text Mozilla/5.0 (Linux; Android 7.0; Nexus 6 Build/NBD90Z) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.124 Mobile Safari/537.36 ``` -------------------------------- ### Example Test Case with Expected Output Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md This example demonstrates a complete test case with a user agent string and its expected parsed fields. It shows how to define new fields and their values. ```yaml - input: user_agent_string: 'Mozilla/5.0 (compatible; Foo/3.1; Bar)' expected: DeviceClass : 'Unknown' DeviceName : 'Unknown' OperatingSystemClass : 'Unknown' OperatingSystemName : 'Unknown' OperatingSystemVersion : '??' LayoutEngineClass : 'Browser' LayoutEngineName : 'Mozilla' LayoutEngineVersion : '5.0' LayoutEngineVersionMajor : '5' LayoutEngineNameVersion : 'Mozilla 5.0' LayoutEngineNameVersionMajor : 'Mozilla 5' AgentClass : 'Browser' AgentName : 'Foo' AgentVersion : '3.1' AgentVersionMajor : '3' AgentNameVersion : 'Foo 3.1' AgentNameVersionMajor : 'Foo 3' MinorFooVersion : '1' ``` -------------------------------- ### Example Useragent String Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md This is an example useragent string that will be used to demonstrate the parse tree and its flattened form. Spaces within the string are preserved for illustrative purposes. ```plaintext foo/1.0 ( one ; two three; four ) bar/2.0 (five;six seven) ``` -------------------------------- ### Chaining Operators Example Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md Demonstrates how to chain operators to navigate and extract specific values from a parsed user agent string. This example shows the progression of the current path and the resulting value after each operator. ```text foo faa/1.0/2.3 (one; two three four) bar baz/2.0/3.0 (five; six seven) ``` -------------------------------- ### Apache HTTPD Accept-CH Header Example Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/ClientHints.md Example of the Accept-CH header that a website should return to inform the browser which Client Hints to send. This header must be sent with every request. ```http Accept-CH: Sec-CH-UA, Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Form-Factors, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-WoW64 ``` -------------------------------- ### Yauaa Servlet Logging Example Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/WebServlet.md Example log output when the Yauaa servlet loads custom rules from a specified directory. ```log Loading 1 rule files using expression: file:UserAgents*/*.yaml - Preparing InternalTraffic.yaml (9608 bytes) - Loaded 1 files in 11 msec using expression: file:UserAgents*/*.yaml ``` -------------------------------- ### Package Quarkus Application for Native Executable Source: https://github.com/nielsbasjes/yauaa/blob/main/analyzer/src/it/Examples/quarkus/README.md Create a native executable for the Quarkus application. Ensure GraalVM is installed. ```bash mvn package -Pnative ``` -------------------------------- ### Start Yauaa Docker Web Servlet Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Commandline.md Pull the Yauaa Docker image and start the webservlet in detached mode, exposing port 8080. ```bash docker pull nielsbasjes/yauaa:{{%YauaaVersion%}} docker run --detach -p8080:8080 nielsbasjes/yauaa:{{%YauaaVersion%}} ``` -------------------------------- ### Install Elasticsearch Plugin (9.x) Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Elastic-Search.md Install the built Yauaa Elasticsearch plugin on your Elasticsearch 9.x instance using the provided file path. ```bash bin/elasticsearch-plugin install file:///path/to/yauaa-elasticsearch-9-{{%YauaaVersion%}}.zip ``` -------------------------------- ### Unit Test Example Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md This snippet shows a unit test configuration, likely used to verify the behavior of the defined matchers and extractors. It includes options for verbosity and initialization. ```yaml - test: options: - 'verbose' - 'init' ``` -------------------------------- ### Hive Table Setup and Data Insertion Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Hive.md Set up a Hive table to store user agent strings and insert sample data for testing the ParseUserAgent UDF. ```sql CREATE TABLE useragents (useragent STRING COMMENT 'The useragent string'); ``` ```sql INSERT INTO TABLE useragents VALUES ('Mozilla/5.0 (Linux\; Android 6.0\; Nexus 6 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36'); ``` ```sql INSERT INTO TABLE useragents VALUES ('Mozilla/5.0 (Linux\; Android 5.1\; Nexus 10 Build/LMY47D) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.109 Safari/537.36'); ``` ```sql INSERT INTO TABLE useragents VALUES ('Mozilla/5.0 (X11\; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'); ``` ```sql INSERT INTO TABLE useragents VALUES ('Mozilla/5.0 (Windows NT 10.0\; Win64\; x64\; Xbox\; Xbox One) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2486.0 Safari/537.36 Edge/13.10553'); ``` -------------------------------- ### Example Apache HTTPD Access Log Lines with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/ClientHints.md Sample log entries from an Apache HTTPD webserver demonstrating how Client Hints are recorded in the access log. These lines show the detailed information captured. ```log 45.138.228.54 - - [02/May/2022:12:25:10 +0200] "GET / HTTP/1.1" 200 16141 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" " Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\"" "\"x86\"" "\"64\"" "-" "\"100.0.4896.127\"" " Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"100.0.4896.127\", \"Google Chrome\";v=\"100.0.4896.127\"" "?0" "\"\"" "\"Linux\"" "\"5.13.0\"" "?0" try.yauaa.basjes.nl 45.138.228.54 - - [02/May/2022:12:25:34 +0200] "GET / HTTP/1.1" 200 15376 "-" "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.0.0 Mobile Safari/537.36" " Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\"" "\"\"" "-" "-" "\"101.0.4951.41\"" " Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"101.0.4951.41\", \"Google Chrome\";v=\"101.0.4951.41\"" "?1" "\"Nokia 7.2\"" "\"Android\"" "\"11.0.0\"" "?0" try.yauaa.basjes.nl ``` -------------------------------- ### Install Elasticsearch Plugin (8.x) Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Elastic-Search.md Install the built Yauaa Elasticsearch plugin on your Elasticsearch 8.x instance using the provided file path. ```bash bin/elasticsearch-plugin install file:///path/to/yauaa-elasticsearch-8-{{%YauaaVersion%}}.zip ``` -------------------------------- ### Install Yauaa Nifi NAR File Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Nifi/_index.md Copy the NAR file to the specified NiFi lib directory to install the ParseUserAgent processor. ```bash cp ./udfs/nifi/nifi-nar/target/yauaa-nifi-.nar /lib ``` -------------------------------- ### Flink SQL UDF Example with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Flink-Table.md An example of calling the ParseUserAgent UDF in Flink SQL, providing specific User-Agent Client Hint headers. ```sql ParseUserAgent( 'User-Agent', useragent, 'Sec-CH-UA-Platform', chPlatform, 'Sec-CH-UA-Platform-Version', chPlatformVersion ) AS parsedUseragent ``` -------------------------------- ### Basic User-Agent Client Hints Parsing with Yauaa Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/ClientHints.md Demonstrates how to initialize the UserAgentAnalyzer and parse request headers including Client Hints. This snippet shows the basic setup for analyzing client information. ```java UserAgentAnalyzer uaa = UserAgentAnalyzer .newBuilder() .build(); Map requestHeaders = new TreeMap<>(); requestHeaders.put("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.0.0 Safari/537.36"); requestHeaders.put("Sec-Ch-Ua", "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\""); requestHeaders.put("Sec-Ch-Ua-Arch", "\"x86\""); requestHeaders.put("Sec-Ch-Ua-Full-Version-List", "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"100.0.4896.75\", \"Google Chrome\";v=\"100.0.4896.75\""); requestHeaders.put("Sec-Ch-Ua-Mobile", "?0"); requestHeaders.put("Sec-Ch-Ua-Model", "\"\""); requestHeaders.put("Sec-Ch-Ua-Platform", "\"Windows\""); requestHeaders.put("Sec-Ch-Ua-Platform-Version", "\"0.1.0\""); requestHeaders.put("Sec-Ch-Ua-Wow64", "?0"); UserAgent userAgent = uaa.parse(requestHeaders); ``` -------------------------------- ### Example User Agent String and Expected Output Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md This snippet shows a sample user agent string and the expected parsed output, illustrating the structure of the data Yauaa produces. ```yaml - "only" input: user_agent_string: 'Mozilla/5.0 (compatible; Foo/3.1; Bar)' expected: DeviceClass : 'Unknown' DeviceName : 'Unknown' OperatingSystemClass : 'Unknown' OperatingSystemName : 'Unknown' OperatingSystemVersion : '??' LayoutEngineClass : 'Browser' LayoutEngineName : 'Mozilla' LayoutEngineVersion : '5.0' LayoutEngineVersionMajor : '5' LayoutEngineNameVersion : 'Mozilla 5.0' LayoutEngineNameVersionMajor : 'Mozilla 5' AgentClass : 'Browser' AgentName : 'Foo' AgentVersion : '3.1' AgentVersionMajor : '3' AgentNameVersion : 'Foo 3.1' AgentNameVersionMajor : 'Foo 3' ``` -------------------------------- ### Install Yauaa Logstash Plugin Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Elastic-Logstash.md Commands to remove the existing plugin and install the Yauaa Logstash filter from a local gem file. ```bash logstash-plugin remove logstash-filter-yauaa logstash-plugin install ./udfs/logstash/target/logstash-filter-yauaa-{{%YauaaVersion%}}.gem ``` -------------------------------- ### Build UserAgentAnalyzer in Scala Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/_index.md Instantiate and configure UserAgentAnalyzer in Scala using the builder pattern. This example demonstrates setting cache size, hiding load statistics, and specifying fields. ```scala val uaa = UserAgentAnalyzer.newBuilder .withCache(10000) .hideMatcherLoadStats .withField("DeviceClass") .build ``` -------------------------------- ### Chaining Operators Table Example Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md Illustrates the effect of various chaining operators on the current path and value extraction from a user agent string. Each row represents a step in the chaining process. ```text agent.product.(1)comments.entry.(1)text[2]="seven" agent.(2)product.(1)comments.(2)entry.(1)text[2] seven ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^ agent.(2)product.(1)comments.(2)entry six seven ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^ agent.(2)product.(1)comments (five; six seven) ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^ agent.(2)product bar baz/2.0/3.0 (five; six seven) ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^< agent.(1)product foo faa/1.0/2.3 (one; two three four) ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name agent.(1)product.(1)name foo faa ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa" agent.(1)product.(1)name foo faa ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^ agent.(1)product foo faa/1.0/2.3 (one; two three four) ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^.comments agent.(1)product.(1)comments (one; two three four) ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^.comments.entry agent.(1)product.(1)comments.(1)entry one ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^.comments.entry.text[2]="three" agent.(1)product.(1)comments.(2)entry[2] three ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^.comments.entry.text[2]="three"@ agent.(1)product.(1)comments.(2)entry two three four ``` ```text agent.product.(1)comments.entry.(1)text[2]="seven"^^^<.name="foo faa"^.comments.entry.text[2]="three"@[1] agent.(1)product.(1)comments.(2)entry[1] two ``` -------------------------------- ### Example of a Samsung Tablet User-Agent Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/expect/Limitations.md This user-agent string was observed in live website logs and demonstrates a case where the analysis might incorrectly classify a device. ```plaintext Mozilla/5.0 (Linux; Android 5.1; SAMSUNG-T805s Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.94 Mobile Safari/537.36 Mozilla/5.0 (Linux; Android 4.4.2; SAMSUNG-T805S Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.89 Safari/537.36 ``` -------------------------------- ### Apache Beam UDF: Extracting Data with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam.md This example shows how to use the UserAgentAnalysisDoFn with Apache Beam when client hints are available. It overrides the getRequestHeaders method to provide a map of request headers, enabling richer analysis. ```java .apply("Extract Elements from Useragent and Client Hints", ParDo.of(new UserAgentAnalysisDoFn(15000) { // Setting the cacheSize @Override public Map getRequestHeaders(TestRecord element) { return element.getHeaders(); } @SuppressWarnings("unused") // Called via the annotation @YauaaField("DeviceClass") public void setDC(TestRecord record, String value) { record.deviceClass = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("AgentNameVersion") public void setANV(TestRecord record, String value) { record.agentNameVersion = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("OperatingSystemNameVersion") public void setOSNV(TestRecord record, String value) { record.operatingSystemNameVersion = value; } }) ) ``` -------------------------------- ### Describe Hive Function Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Hive.md Verify the installation and basic signature of the ParseUserAgent Hive function. ```sql DESCRIBE FUNCTION ParseUserAgent; ``` -------------------------------- ### Kubernetes ConfigMap for Custom Rules Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/Kubernetes.md Define custom user-agent rules and store them in a Kubernetes ConfigMap. This example shows a basic rule and a test case. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: niels-yauaa-rules namespace: yauaa data: MyCustomRules.yaml: | config: - matcher: require: - 'agent.product.name="NielsBasjes"' extract: - 'CustomRuleDemonstrationName : 42 :"A Simple demonstration of a custom rule"' - 'CustomRuleDemonstrationWebsite : 42 :"https://yauaa.basjes.nl"' - test: input: user_agent_string: 'NielsBasjes/42 (https://niels.basjes.nl)' expected: DeviceClass : 'Robot' DeviceName : 'Basjes Robot' DeviceBrand : 'Basjes' OperatingSystemClass : 'Cloud' OperatingSystemName : 'Cloud' OperatingSystemVersion : '??' OperatingSystemVersionMajor : '??' OperatingSystemNameVersion : 'Cloud ??' OperatingSystemNameVersionMajor : 'Cloud ??' LayoutEngineClass : 'Unknown' LayoutEngineName : 'Unknown' LayoutEngineVersion : '??' LayoutEngineVersionMajor : '??' LayoutEngineNameVersion : 'Unknown ??' LayoutEngineNameVersionMajor : 'Unknown ??' AgentClass : 'Special' AgentName : 'NielsBasjes' AgentVersion : '42' AgentVersionMajor : '42' AgentNameVersion : 'NielsBasjes 42' AgentNameVersionMajor : 'NielsBasjes 42' AgentInformationUrl : 'https://niels.basjes.nl' CustomRuleDemonstrationName : 'A Simple demonstration of a custom rule' CustomRuleDemonstrationWebsite : 'https://yauaa.basjes.nl' ``` -------------------------------- ### Analyze User Agent String Output Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Commandline.md Example YAML output from analyzing a user agent string with the Yauaa webservlet. ```yaml - test: input: user_agent_string: 'Mozilla/5.0 (Linux; Android 7.0; Nexus 6 Build/NBD90Z) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.124 Mobile Safari/537.36' expected: DeviceClass : 'Phone' DeviceName : 'Google Nexus 6' DeviceBrand : 'Google' OperatingSystemClass : 'Mobile' OperatingSystemName : 'Android' OperatingSystemVersion : '7.0' OperatingSystemVersionMajor : '7' OperatingSystemNameVersion : 'Android 7.0' OperatingSystemNameVersionMajor : 'Android 7' OperatingSystemVersionBuild : 'NBD90Z' LayoutEngineClass : 'Browser' LayoutEngineName : 'Blink' LayoutEngineVersion : '53.0' LayoutEngineVersionMajor : '53' LayoutEngineNameVersion : 'Blink 53.0' LayoutEngineNameVersionMajor : 'Blink 53' AgentClass : 'Browser' AgentName : 'Chrome' AgentVersion : '53.0.2785.124' AgentVersionMajor : '53' AgentNameVersion : 'Chrome 53.0.2785.124' AgentNameVersionMajor : 'Chrome 53' ``` -------------------------------- ### Package Quarkus Application for Native Executable (Container) Source: https://github.com/nielsbasjes/yauaa/blob/main/analyzer/src/it/Examples/quarkus/README.md Create a native executable for the Quarkus application using a containerized build process. This is useful if GraalVM is not installed locally. ```bash mvn clean package -Pnative -Dquarkus.native.container-build=true ``` -------------------------------- ### Example Test Failure Output Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md This output illustrates a test failure scenario, highlighting missing fields ('<>') and discrepancies between expected and actual values. It's useful for debugging new rules. ```text [INFO ] AbstractUserAgentAnalyzerTester : 89: +--------+------------------------------+-------------+---------+------------+------------+ [INFO ] AbstractUserAgentAnalyzerTester : 89: | Result | Field | Actual | Default | Confidence | Expected | [INFO ] AbstractUserAgentAnalyzerTester : 89: +--------+------------------------------+-------------+---------+------------+------------+ [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineClass | Browser | | 3 | <> | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineName | Mozilla | | 3 | <> | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineVersion | 5.0 | | 3 | <> | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineVersionMajor | 5 | | 3 | <> | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineNameVersion | Mozilla 5.0 | | 3 | <> | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | LayoutEngineNameVersionMajor | Mozilla 5 | | 3 | <> | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentClass | Browser | | 10 | | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentName | Foo | | 10 | | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentVersion | 3.1 | | 10 | | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentVersionMajor | 3 | | 10 | | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentNameVersion | Foo 3.1 | | 10 | | [INFO ] AbstractUserAgentAnalyzerTester : 89: | | AgentNameVersionMajor | Foo 3 | | 10 | | [ERROR] AbstractUserAgentAnalyzerTester : 109: | -FAIL- | AgentLanguage | Bavarian | | 500006 | <> | ``` -------------------------------- ### Flink UDF: Parse User-Agent with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Flink.md Example of implementing UserAgentAnalysisMapper to parse user agent information including Client Hints. Override getRequestHeaders to provide the necessary header map. ```java Map requestHeaders = new TreeMap<>(); requestHeaders.put("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.0.0 Safari/537.36"); requestHeaders.put("Sec-Ch-Ua", "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"100\", \"Google Chrome\";v=\"100\" "); requestHeaders.put("Sec-Ch-Ua-Arch", "\"x86\""); requestHeaders.put("Sec-Ch-Ua-Full-Version-List", "\" Not A;Brand\";v=\"99.0.0.0\", \"Chromium\";v=\"100.0.4896.75\", \"Google Chrome\";v=\"100.0.4896.75\" "); requestHeaders.put("Sec-Ch-Ua-Mobile", "?0"); requestHeaders.put("Sec-Ch-Ua-Model", "\"\""); requestHeaders.put("Sec-Ch-Ua-Platform", "\"Windows\""); requestHeaders.put("Sec-Ch-Ua-Platform-Version", "\"0.1.0\""); requestHeaders.put("Sec-Ch-Ua-Wow64", "?0"); ``` ```java .map(new UserAgentAnalysisMapper(15000) { // Setting the cacheSize @Override public Map getRequestHeaders(TestRecord element) { return element.getHeaders(); } @SuppressWarnings("unused") // Called via the annotation @YauaaField("DeviceClass") public void setDC(TestRecord record, String value) { record.deviceClass = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("AgentNameVersion") public void setANV(TestRecord record, String value) { record.agentNameVersion = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("OperatingSystemNameVersion") public void setOSNV(TestRecord record, String value) { record.operatingSystemNameVersion = value; } }) ``` -------------------------------- ### Example Apache Pig Script with Yauaa UDF Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Pig.md This script demonstrates how to register the Yauaa UDF, define it with optional parameters for cache size and specific fields, and then use it to parse user agent strings from raw data. ```pig -- Import the UDF jar file so this script can use it REGISTER ../target/*-udf.jar; ------------------------------------------------------------------------ -- Define a more readable name for the UDF and pass optional parameters -- First parameter is ALWAYS the cache size (as a text string!) -- The parameters after that are the requested fields. ---------- -- If you simply want 'everything' -- DEFINE ParseUserAgent nl.basjes.parse.useragent.pig.ParseUserAgent; ---------- -- If you just want to set the cache -- DEFINE ParseUserAgent nl.basjes.parse.useragent.pig.ParseUserAgent('10000'); ---------- -- If you want to set the cache and only retrieve the specified fields DEFINE ParseUserAgent nl.basjes.parse.useragent.pig.ParseUserAgent('10000', 'DeviceClass', 'DeviceBrand' ); rawData = LOAD 'testcases.txt' USING PigStorage() AS ( useragent: chararray ); UaData = FOREACH rawData GENERATE useragent, -- Do NOT specify a type for this field as the UDF provides the definitions ParseUserAgent(useragent) AS parsedAgent; ``` -------------------------------- ### Aggregate Parsed User Agent Fields Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Drill.md This example demonstrates how to parse user agent strings from a file, extract specific fields from the resulting Drill map, and then aggregate these fields. It requires the user agent strings to be in the first column of a TSV file. ```sql SELECT uadata.ua.AgentNameVersion AS Browser, COUNT( * ) AS BrowserCount FROM ( SELECT parse_user_agent( columns[0] ) AS ua FROM dfs.`/tmp/testcase.tsv` ) AS uadata GROUP BY uadata.ua.AgentNameVersion ORDER BY BrowserCount DESC; ``` -------------------------------- ### Describe Extended Hive Function Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Hive.md Get detailed information about the ParseUserAgent Hive function, including its synonyms and an example of its usage. ```sql > DESCRIBE FUNCTION EXTENDED ParseUserAgent; ``` -------------------------------- ### Comparing Values: Starts With Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md Check if a node's value starts with a specific prefix. Returns the first matching node during backtracking that starts with the prefix. ```Yauaa Rule Language agent.product.name{"b" ``` -------------------------------- ### Matcher with Performance Speedup Trick Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/developer/MakingNewRules.md Demonstrates a matcher that uses a direct path lookup for a specific version as a speedup trick. If this direct check fails, other more complex checks and lookups are not attempted, improving performance. ```yaml - matcher: require: - 'agent.product.(1)name="Chrome"' - 'LookUp[ChromeLayoutEngineName;agent.product.(1)name="Chrome"^.version[1]]' - 'agent.product.(1)name="AppleWebKit"^.version[1]="537"' - 'agent.product.(1)version[1]="537" # This is a matching speedup trick extract: - 'LayoutEngineClass : 1000:"Browser"' - 'LayoutEngineName : 1000:"AppleWebKit"' - 'LayoutEngineVersion : 1000:agent.product.(1)name="AppleWebKit"^.version' ``` -------------------------------- ### Combine Specific Fields and Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md Parse user agent data by requesting specific fields and providing Client Hints headers simultaneously. The order of parameters does not matter. ```sql ParseUserAgent( 'user-Agent', userAgent, 'sec-CH-UA-Platform', chPlatform, 'sec-CH-UA-Platform-Version', chPlatformVersion 'DeviceClass', 'AgentNameVersionMajor', 'OperatingSystemNameVersion', ) ``` -------------------------------- ### Parse User Agent with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Drill.md Parses a User-Agent string along with Client Hints headers and their values. This allows for more detailed analysis by providing additional context. ```sql parse_user_agent ( , [
,]+ ) ``` -------------------------------- ### Older iPad User-Agents Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/expect/Limitations.md Examples of user-agent strings previously reported by iPads. ```plaintext Mozilla/5.0 (iPad; CPU OS 18_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 Mozilla/5.0 (iPad; CPU OS 18_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) ``` -------------------------------- ### Google Charts Initialization and Resize Handling Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/layouts/shortcodes/PerformanceGraph.html Loads the Google Charts library and sets up event listeners for window resizing to redraw the chart. This ensures the chart remains responsive. ```javascript google.charts.load('current', {'packages':['corechart']}); google.charts.setOnLoadCallback(drawChart); //create trigger to resizeEnd event $(window).resize(function() { if(this.resizeTO) clearTimeout(this.resizeTO); this.resizeTO = setTimeout(function() { $(this).trigger('resizeEnd'); }, 500); }); //redraw graph when window resize is completed $(window).on('resizeEnd', function() { drawChart(); }); ``` -------------------------------- ### Apache HTTPD LogFormat for Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/ClientHints.md Configure Apache's LogFormat directive to include various Client Hints in the access logs. This allows for detailed logging of client information. ```apache LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Sec-CH-UA}i\" \"%{Sec-CH-UA-Arch}i\" \"%{Sec-CH-UA-Bitness}i\" \"%{Sec-CH-UA-Form-Factors}i\" \"%{Sec-CH-UA-Full-Version}i\" \"%{Sec-CH-UA-Full-Version-List}i\" \"%{Sec-CH-UA-Mobile}i\" \"%{Sec-CH-UA-Model}i\" \"%{Sec-CH-UA-Platform}i\" \"%{Sec-CH-UA-Platform-Version}i\" \"%{Sec-CH-UA-WoW64}i" %V" combinedhintsvhost ``` -------------------------------- ### Parse User Agent with Client Hints (Alternative Format) Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Snowflake/_index.md An alternative method to parse User-Agent strings and Client Hints in Snowflake, where the User-Agent string is provided as the first element in the array. ```sql select parse_useragent( ['Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36', 'Sec-Ch-Ua', '" Not A;Brand";v="99", "Chromium";v="100", "Google Chrome";v="100"', 'Sec-Ch-Ua-Arch', '"x86"', 'Sec-Ch-Ua-Bitness', '"64"', 'Sec-Ch-Ua-Full-Version', '"100.0.4896.127"', 'Sec-Ch-Ua-Full-Version-List', '" Not A;Brand";v="99.0.0.0", "Chromium";v="100.0.4896.127", "Google Chrome";v="100.0.4896.127"', 'Sec-Ch-Ua-Mobile', '?0', 'Sec-Ch-Ua-Model', '""', 'Sec-Ch-Ua-Platform', '"Linux"', 'Sec-Ch-Ua-Platform-Version', '"5.13.0"', 'Sec-Ch-Ua-Wow64', '?0'] ) as ua_obj, ua_obj:OperatingSystemNameVersion::string as operating_system_name_version; ``` -------------------------------- ### Kubernetes Ingress Configuration Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/Kubernetes.md Expose the Yauaa service outside the cluster using an Ingress resource. This example uses nginx-ingress. ```yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: yauaa namespace: yauaa annotations: kubernetes.io/ingress.class: nginx spec: rules: - host: yauaa.example.nl http: paths: - path: / pathType: Prefix backend: service: name: yauaa port: number: 80 ``` -------------------------------- ### Get Specific Fields as a Map Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md Extract only a subset of desired fields from the user agent string using the ParseUserAgent function. ```sql ParseUserAgent(userAgent, 'DeviceClass', 'AgentNameVersion') AS someFields ``` -------------------------------- ### Run Native Executable of Quarkus Application Source: https://github.com/nielsbasjes/yauaa/blob/main/analyzer/src/it/Examples/quarkus/README.md Execute the compiled native binary of the Quarkus application. ```bash ./target/yauaa-example-quarkus-*-runner ``` -------------------------------- ### Register DataStream in Flink Table Environment Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Flink-Table.md Example of registering a DataStream as a table in Flink's TableEnvironment, specifying column names. ```java // Give the stream a Table Name tableEnv.registerDataStream("AgentStream", inputStream, "timestamp, url, useragent"); ``` -------------------------------- ### Get a Single Field from User Agent String Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md Use the ParseUserAgentField function to extract a specific field, such as 'DeviceClass', from a user agent string. ```sql ParseUserAgentField(userAgent, 'DeviceClass') AS deviceClassField ``` -------------------------------- ### Build Yauaa Software Source: https://github.com/nielsbasjes/yauaa/blob/main/webapp/README.md Builds the entire Yauaa software project. Ensure you are in the parent directory before running. ```bash (cd .. && mvn clean package ) ``` -------------------------------- ### Extract a Specific Field from a Map Result Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md After obtaining a map of fields, use SQL syntax to extract a specific field, for example, 'DeviceClass' from the map. ```sql ParseUserAgent(userAgent, 'DeviceClass')['DeviceClass'] AS deviceClass ``` -------------------------------- ### Instantiate UserAgentAnalyzer Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/_index.md Create an instance of UserAgentAnalyzer with specific configurations like hiding load stats and setting cache size. Avoid frequent instantiation due to high initialization cost. ```java UserAgentAnalyzer uaa = UserAgentAnalyzer .newBuilder() .hideMatcherLoadStats() .withCache(10000) .build(); ``` -------------------------------- ### Set Elasticsearch Version in pom.xml Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Elastic-Search.md Modify the pom.xml file to specify the target Elasticsearch version for the plugin build. This ensures compatibility with your Elasticsearch installation. ```bash sed -i 's@[^<]\+@8.17.3@g' pom.xml ``` -------------------------------- ### Apache Drill SQL Query with User-Agent Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Drill.md This SQL query analyzes web server logs from a specified file, extracting detailed user-agent information including Client Hints. It uses a custom log format to capture various Client Hints headers. ```sql SELECT deviceClass, agentNameVersionMajor, operatingSystemNameVersion FROM table( dfs.`/tmp/access.hints` ( type => 'httpd', logFormat => '%a %l %u %t "%r" %>s %b "%%{Referer}i" "%%{User-Agent}i" "%%{Sec-CH-UA}i" "%%{Sec-CH-UA-Arch}i" "%%{Sec-CH-UA-Bitness}i" "%%{Sec-CH-UA-Full-Version}i" "%%{Sec-CH-UA-Full-Version-List}i" "%%{Sec-CH-UA-Mobile}i" "%%{Sec-CH-UA-Model}i" "%%{Sec-CH-UA-Platform}i" "%%{Sec-CH-UA-Platform-Version}i" "%%{Sec-CH-UA-WoW64}i" %V', flattenWildcards => true ) ) AS uadata; ``` -------------------------------- ### Parse User Agent with Client Hints Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md Demonstrates using ParseUserAgentJson with Client Hints headers to extract specific user agent information. This method allows for dynamic retrieval of fields based on available client hints. ```Java ParseUserAgentJson( 'DeviceClass', 'user-Agent', userAgent, 'AgentNameVersionMajor', 'sec-CH-UA-Platform', chPlatform, 'OperatingSystemNameVersion', 'sec-CH-UA-Platform-Version', chPlatformVersion ) ``` -------------------------------- ### Logstash Output with Yauaa Fields Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Elastic-Logstash.md Example of Logstash output after the yauaa filter has processed a record. It shows the original fields and the newly added fields extracted by Yauaa. ```json { "@timestamp" => 2022-04-24T09:30:11.051Z, "@version" => "1", "host" => "b9bf088b8c92", "message" => "HeartBeat event to trigger the test", "ua" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36", "uap" => "\"macOS\"", "uapv" => "\"12.3.1\"", "uaDC" => "Desktop", "uaDN" => "Apple Macintosh", "uaDB" => "Apple", "uaOSC" => "Desktop", "uaOSNV" => "Mac OS 12.3.1", "uaLEC" => "Browser", "uaLENV" => "Blink 100.0", "uaAC" => "Browser", "uaAN" => "Chrome", "uaANVM" => "Chrome 100", "uaAANV" => "Chrome 100.0.4896.60" } ``` -------------------------------- ### Flink UDF: Parse Only User-Agent String Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Flink.md Example of implementing the UserAgentAnalysisMapper to parse a user agent string from a record. The @YauaaField annotation maps parsed values to record fields. ```java .map(new UserAgentAnalysisMapper(15000) { // Setting the cacheSize @Override public String getUserAgentString(TestRecord record) { return record.useragent; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("DeviceClass") public void setDC(TestRecord record, String value) { record.deviceClass = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("AgentNameVersion") public void setANV(TestRecord record, String value) { record.agentNameVersion = value; } @SuppressWarnings("unused") // Called via the annotation @YauaaField("OperatingSystemNameVersion") public void setOSNV(TestRecord record, String value) { record.operatingSystemNameVersion = value; } }) ``` -------------------------------- ### Combine Specific Fields and Client Hints (Shuffled Order) Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Beam-Sql.md Demonstrates that the order of requesting specific fields and Client Hints headers can be shuffled in the ParseUserAgent function call. ```sql ParseUserAgent( 'DeviceClass', 'user-Agent', userAgent, 'AgentNameVersionMajor', 'sec-CH-UA-Platform', chPlatform, 'OperatingSystemNameVersion', 'sec-CH-UA-Platform-Version', chPlatformVersion ) ``` -------------------------------- ### Flink SQL Query Extracting Specific Fields Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/udf/Apache-Flink-Table.md Example of a Flink SQL query that parses the useragent and then extracts specific fields like 'DeviceClass' and 'AgentNameVersionMajor' from the resulting map. ```java // Register the function with all the desired fieldnames and optionally the size of the cache tableEnv.registerFunction("ParseUserAgent", new AnalyzeUseragentFunction(15000, "DeviceClass", "AgentNameVersionMajor")); // Define the query. String sqlQuery = "SELECT useragent,"+ " parsedUseragent['DeviceClass'] AS deviceClass," + " parsedUseragent['AgentNameVersionMajor'] AS agentNameVersionMajor " + "FROM ( " + " SELECT useragent," + " ParseUserAgent(useragent) AS parsedUseragent" + " FROM AgentStream " + ")"; Table resultTable = tableEnv.sqlQuery(sqlQuery); // 3 Strings TypeInformation tupleType = new RowTypeInfo(STRING, STRING, STRING); DataStream resultSet = tableEnv.toAppendStream(resultTable, tupleType); ``` -------------------------------- ### Parse User Agent String and Get Field Values Source: https://github.com/nielsbasjes/yauaa/blob/main/documentation/content/using/_index.md Use the 'parse' method to analyze a user agent string and then iterate through the available field names to retrieve their values. ```java UserAgent agent = uaa.parse("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11"); for (String fieldName: agent.getAvailableFieldNamesSorted()) { System.out.println(fieldName + " = " + agent.getValue(fieldName)); } ```