### Install and Start QLever Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines Installs QLever using pip, indexes data, starts the server, and prepares for querying. This is the primary tool for performance evaluation. ```shell pip install qlever qlever index sudo bash -c "sync; sleep 5; echo 3 > /proc/sys/vm/drop_caches" qlever start qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:7015 ``` -------------------------------- ### Setup QLever for PubChem Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-PubChem Initial setup commands for QLever to configure, download, index, and start the PubChem dataset. ```bash qlever setup-config pubchem qlever get-data qlever index qlever start qlever ui ``` -------------------------------- ### Setup Wikidata Configuration and Data Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-Wikidata These commands set up the QLever configuration for Wikidata, download the dataset, build the index, and start the server and UI. ```bash mkdir wikidata && cd wikidata qlever setup-config wikidata qlever get-data qlever index qlever start qlever ui ``` -------------------------------- ### Install and Configure Stardog Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines Installs Stardog, configures its memory settings, creates a database, and starts the server. Use for bulk loading data. ```shell sudo apt install gnupg curl http://packages.stardog.com/stardog.gpg.pub | sudo apt-key add echo "deb http://packages.stardog.com/deb/ stable main" | sudo tee -a /etc/apt/sources.list sudo apt-get update sudo apt-get install -y stardog=10.0.1 sudo apt-get install bash-completion source /opt/stardog/bin/stardog-completion.sh sudo sed -i 's/UseParallelOldGC/UseParallelGC/' /opt/stardog/bin/helpers.sh export PATH=$PATH:/opt/stardog/bin/ export STARDOG_SERVER_JAVA_ARGS="-Xms20g -Xmx20g -XX:MaxDirectMemorySize=20G -Djava.io.tmpdir=\"$(pwd)\"" export STARDOG_PROPERTIES="$(pwd)/stardog.properties" && echo "memory.mode = bulk_load" > ${STARDOG_PROPERTIES} stardog-admin server start stardog-admin db create -n dblp dblp.ttl.gz stardog-admin server stop echo "memory.mode = read_optimized" > ${STARDOG_PROPERTIES} stardog-admin server start --disable-security stardog-admin metadata set -o query.all.graphs=true -- dblp sudo bash -c "sync; sleep 5; echo 3 > /proc/sys/vm/drop_caches" qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:5820/dblp/query ``` -------------------------------- ### Start QLever Server Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-UniProt Starts the QLever server in the current directory. The server is typically available within 1.5 minutes. ```bash qlever start ``` -------------------------------- ### Apache Jena Setup Commands Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines These commands download and extract Apache Jena components, install Java, and configure it. They prepare the environment for loading and serving the DBLP dataset. ```shell wget https://dlcdn.apache.org/jena/binaries/apache-jena-fuseki-5.0.0.zip unzip apache-jena-fuseki-5.0.0.zip && rm -f $_ wget https://dlcdn.apache.org/jena/binaries/apache-jena-5.0.0.zip unzip apache-jena-5.0.0.zip && rm -f $_ sudo apt update && sudo apt install -y openjdk-21-jdk sudo update-alternatives --config java -> select JDK 21 (auto mode) ``` -------------------------------- ### Install and Configure Virtuoso Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines Clones Virtuoso, installs dependencies, configures the server settings, loads data using isql, and starts the SPARQL endpoint. ```shell git clone https://github.com/openlink/virtuoso-opensource virtuoso sudo apt install -y autoconf automake libtool flex bison gperf gawk m4 make openssl libsrc/Wi/sparql_io.sql -> change maxrows := 1024*1024: to 2*1024*1024 -2; [see https://github.com/openlink/virtuoso-opensource/issues/700] ./autogen.sh && ./configure && make && sudo make install virtuoso.ini -> change: ServerPort = 8888, NumberOfBuffers and MaxDirtyBuffers = presets for 64 GB free, DefaultHost = hostname:8890, DirsAllowed = directory with the input files, ResultsSetMaxRows = 2000000, MaxQueryCostEstimationTime = 3600, MaxQueryExecutionTime = 3600, MaxQueryMem = 20G isql-vt 8888 SQL> ld_dir('/local/data/qlever/qlever-indices/virtuoso-playground.ssd', 'dblp.ttl.gz', ''); SQL> DB.DBA.rdf_loader_run(); SQL> checkpoint; SQL> exit; sudo bash -c "sync; sleep 5; echo 3 > /proc/sys/vm/drop_caches" /usr/bin/virtuoso-t -f & qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:8890/sparql ``` -------------------------------- ### Start QLever Server Source: https://github.com/ad-freiburg/qlever/wiki/First-tests-with-SPARQL-1.1-Update Starts the QLever server, ensuring no other process is using the same port. This is a prerequisite for performing update and query operations. ```bash qlever start --kill-existing-with-same-port ``` -------------------------------- ### Install and Configure Blazegraph Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines Downloads Blazegraph JAR, starts the server, loads data in chunks using curl, and prepares for querying. Suitable for large datasets. ```shell wget https://github.com/blazegraph/database/releases/download/BLAZEGRAPH_2_1_6_RC/blazegraph.jar java -server -Xmx20g -jar blazegraph.jar & docker run -it --rm -v $(pwd):/data stain/jena riot --output=NT /data/dblp.ttl.gz | split -a 3 --numeric-suffixes=1 --additional-suffix=.nt -l 1000000 --filter='gzip > $FILE.gz' - dblp- for CHUNK in dblp-???.nt.gz; do curl -s indus:9999/blazegraph/namespace/kb/sparql --data-binary update="LOAD "; done kill %1 sudo bash -c "sync; sleep 5; echo 3 > /proc/sys/vm/drop_caches" java -server -Xmx20g -jar blazegraph.jar & qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:9999/blazegraph/namespace/kb/sparql ``` -------------------------------- ### Install and Configure GraphDB Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines Downloads, unzips, and configures GraphDB. Creates a database, imports data, and prepares for querying. Use for read-optimized operations. ```shell Fill out form on https://www.ontotext.com/products/graphdb/download/ Click on link "Platform-independent distribution" in mail and download graphdb-10.6.2-dist.zip unzip graphdb-10.6.2-dist.zip && rm -f $_ graphdb-10.6.2/bin/console > create graphdb [ID = dblp, rest = default] > quit graphdb-10.6.2/bin/importrdf preload -f -i dblp dblp.ttl.gz graphdb-10.6.2/bin/graphdb sudo bash -c "sync; sleep 5; echo 3 > /proc/sys/vm/drop_caches" curl -s localhost:7200/repositories/dblp --data-urlencode 'query=SELECT * { ?s ?p ?o } LIMIT 1' [minimal warmup] qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:7200/repositories/dblp ``` -------------------------------- ### Apache Jena Fuseki Server Start Command Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines This command starts the Apache Jena Fuseki server to serve the DBLP dataset. It configures the port, data location, and the dataset endpoint. ```shell java -jar apache-jena-fuseki-5.0.0/fuseki-server.jar --port 8015 --loc data /dblp ``` -------------------------------- ### Run QLever Benchmark Queries Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-PubChem Executes 16 example queries against a local QLever SPARQL endpoint, downloading results. This is used for benchmarking QLever's performance. ```bash qlever example-queries --query-ids 1-16 --download-or-count download --sparql-endpoint localhost:7023 ``` -------------------------------- ### Install ANTLR4 Source: https://github.com/ad-freiburg/qlever/blob/master/src/util/ConfigManager/generated/README.md Installs ANTLR4 and its prerequisites using pip. This command creates necessary aliases for using ANTLR4 on your machine. ```bash pip install antlr4-tools ``` -------------------------------- ### Wikidata Settings Configuration Source: https://github.com/ad-freiburg/qlever/wiki/Internal-and-external-vocabulary Example settings.json for Wikidata, specifying internal languages and external prefixes. ```json { "languages-internal": ["en"], "prefixes-external": [ "/target/release oxigraph load -f dblp.ttl.gz -l . ``` -------------------------------- ### UniProt Settings Configuration Source: https://github.com/ad-freiburg/qlever/wiki/Internal-and-external-vocabulary Example settings.json for UniProt, detailing internal languages and a comprehensive list of external prefixes. ```json "languages-internal": ["en"], "prefixes-external": [ " ``` -------------------------------- ### Oxigraph Serve-Read-Only Command Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines This command starts an Oxigraph read-only server after loading data. It's used to make the loaded DBLP dataset available for SPARQL queries at a specified address. ```shell oxigraph serve-read-only -l . -b localhost:8015 ``` -------------------------------- ### Querying Compound Molecular Weight Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-PubChem Example of how to query for the molecular weight of a compound using generic and specific 'has attribute' and 'has value' predicates. ```sparql ?compound sio:SIO_000008 [ rdf:type sio:CHEMINF_000334 ; sio:SIO_000300 ?value ] ``` -------------------------------- ### Configure Abseil C++ Library Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Declares the Abseil C++ library as a Git submodule dependency. It sets installation enablement, system include usage, and C++ standard propagation. ```cmake set(ABSL_ENABLE_INSTALL ON) set(ABSL_USE_SYSTEM_INCLUDES ON) set(ABSL_PROPAGATE_CXX_STD ON) FetchContent_Declare( abseil GIT_REPOSITORY https://github.com/abseil/abseil-cpp.git GIT_TAG 93ac3a4f9ee7792af399cebd873ee99ce15aed08 # 2024-05-16 EXCLUDE_FROM_ALL ) ``` -------------------------------- ### Enable Perftools Profiler Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Adds the `-lprofiler` flag to C++ compiler flags if the `PERFTOOLS_PROFILER` variable is set. This requires google-perftools to be installed. ```cmake if (${PERFTOOLS_PROFILER}) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -lprofiler") message(STATUS "Adding -lprofiler (make sure your have google-perftools installed.)") endif () ``` -------------------------------- ### Include fsst Source Directory Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Adds the source directory of the fsst library to the include paths. This is necessary if fsst headers are not installed in a standard location. ```cmake include_directories(SYSTEM ${fsst_SOURCE_DIR}) ``` -------------------------------- ### Clone QLever and Apply UniProt Settings Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-UniProt Clones the QLever repository, merges a specific PR for UniProt settings, and builds a Docker image. Adjusts RAM limits and predicate storage. ```bash git clone --recursive git@github.com:ad-freiburg/qlever cd qlever git merge origin/uniprot-settings docker build -t qlever.uniprot . ``` -------------------------------- ### Build QLever Index for UniProt Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-UniProt Builds the QLever index for UniProt data using the qlever script and a preconfigured Qleverfile. Downloads the Qleverfile and initiates the index build. ```bash . qlever uniprot qlever index ``` -------------------------------- ### Implementing runAllBenchmarks for a Benchmark Class Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/Usage.md This function demonstrates how to measure function execution times, add measurements to results, create and manipulate tables, and organize measurements into groups. It requires including Benchmark.h and using the ad_benchmark namespace. ```c++ BenchmarkResults runAllBenchmarks(){ BenchmarkResults results {}; /* The functions for measuring the execution time of functions, only take lambdas, which will be called without any arguments. Simply wrap the actual function call, you want to measure. */ auto& dummyFunctionToMeasure = [](){ // Do whatever you might want to measure. }; // In order to recognise later, which time belongs to which benchmark, // most of the functions concerning the measurement of functions, or their // organization, require an identifier. A.k.a. a name. const std::string identifier = "Some identifier"; // Just saves the measured execution time. results.addMeasurement(identifier, dummyFunctionToMeasure); /* Create an empty table with a number of rows and columns. Doesn't measure anything. The number of columns can not be changed after creation, but the number of rows can. Important: The row names aren't saved in a separate container, but INSIDE the first column of the table. */ auto& table = results.addTable(identifier, {"rowName1", "rowName2", "etc."}, {"Column for row names", "columnName1", "columnName2", "etc."}); // You can add measurements to the table as entries, but you can also // read and set entries. table.addMeasurement(0, 2, dummyFunctionToMeasure); // Row 0, column 1. table.setEntry(0, 1, "A custom entry can be any type in 'ad_benchmark::ResultTable::EntryType', except 'std::monostate'."); table.getEntry(0, 2); // The measured time of the dummy function. // Replacing a row name. table.setEntry(0, 0, "rowName1++"); /* Creates an empty group. Doesn't measure anything, but groups and tables can be added to better organize them. */ auto& group = results.addGroup(identifier); // Normal measurement. group.addMeasurement(identifier, dummyFunctionToMeasure); // Table. group.addTable(identifier, {"rowName1", "rowName2", "etc."}, {"Column for row names", "columnName1", "columnName2", "etc."}); return results; } ``` -------------------------------- ### Configure Single Test Binary Build Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Sets up a single executable for all tests when the SINGLE_TEST_BINARY option is enabled. Links against necessary libraries and adds a single test command. ```cmake if (SINGLE_TEST_BINARY) message(STATUS "All tests are linked into a single executable `QLeverAllUnitTestsMain`") add_executable(QLeverAllUnitTestsMain) qlever_target_link_libraries(QLeverAllUnitTestsMain gtest gmock_main testUtil ${CMAKE_THREAD_LIBS_INIT}) # The following line (as opposed to using `gtest_discover_tests`) has the effect that `ctest` will treat all the # unit tests as a single test case and will simply run the executable (using the main function from `gtest_main`. # This decreases the runtime overhead per unit tests in particular for sanitizer builds. add_test(NAME QLeverAllUnitTests COMMAND $) else () message(STATUS "The tests are split over multiple binaries") endif () ``` -------------------------------- ### Defining Concepts with CPP_concept Source: https://github.com/ad-freiburg/qlever/wiki/Porting-QLever-to--Cpp17 Demonstrates how to define concepts using the CPP_concept macro, showing equivalent C++20 and C++17 syntax. ```cpp // Rewrite as follows: template CPP_concept small = sizeof(T) < 3; // This is equivalent in C++20 mode to template concept small = sizeof(T) < 3; // And in C++17 mode to template static constexpr bool small = sizeof(T) < 3; // NOTE: As soon as you rewrite one of the concepts using `CPP_concept` , // then you can't use it in `variant A` of the `CPP_template` example // above anymore, but have to rewrite the template. ``` -------------------------------- ### Run Performance Test Script Source: https://github.com/ad-freiburg/qlever/blob/master/misc/README.md Execute the performance comparison script with a FreebaseEasy index and a specified query file. Output is formatted with 'column' and piped to 'less'. ```bash ./compare_performance_only_own.py \ --index /wikipedia-freebase-easy \ --queryfile query-sets/query-file-notext.txt \ qlever-version-name ../build/SparqlEngineMain cost_factors.default.tsv \ | column -t -s $' ' | less -S ``` -------------------------------- ### Get Original Size of wdt:P31 Source: https://github.com/ad-freiburg/qlever/wiki/First-tests-with-SPARQL-1.1-Update Queries the original size of the wdt:P31 predicate in the Wikidata dataset. Clears the cache before execution and measures the time taken. Assumes the server is running and accessible. ```bash qlever clear-cache 2> /dev/null && /usr/bin/time -f "Elapsed time: %es" curl -s https://qlever.cs.uni-freiburg.de/api/wikidata-prut -H "Accept: application/qlever-results+json" -d "send=0" --data-urlencode "query=PREFIX wdt: SELECT ?s ?o WHERE { ?s wdt:P31 ?o }" | jq .resultsize | numfmt --grouping ``` -------------------------------- ### Define Benchmark Library with Main Function Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/CMakeLists.txt Creates a library that includes the benchmark infrastructure and a default main function. This is useful for creating self-contained benchmark executables. ```cmake add_library(benchmarkWithMain infrastructure/BenchmarkMain.cpp) qlever_target_link_libraries(benchmarkWithMain benchmark) ``` -------------------------------- ### Query Results in QLever JSON Format Source: https://github.com/ad-freiburg/qlever/wiki/QLever-API-documentation Use this curl command to fetch query results in the compact `application/qlever-results+json` format. This format includes additional metadata like query execution time and plan. ```bash curl -s https://qlever.dev/api/wikidata -H "Accept: application/qlever-results+json" -H "Content-type: application/sparql-query" --data "SELECT * WHERE { ?s ?p ?o } LIMIT 1" | jq ``` -------------------------------- ### Link and Discover Tests Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Links a test binary and then uses gtest_discover_tests to add test cases from the executable. Includes a discovery timeout and labels the test. ```cmake function(linkAndDiscoverTest basename) linkTest(${basename} ${ARGN}) gtest_discover_tests(${basename} ${basename} DISCOVERY_TIMEOUT 600 PROPERTIES LABELS "${basename}") endfunction() ``` -------------------------------- ### Add and Link Test (with testUtil) Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Adds a GTest/GMock test case, compiling from basename.cpp. Links against gmock_main, threading library, and testUtil (all of QLever). ```cmake function(addLinkAndDiscoverTest basename) addLinkAndDiscoverTestImpl(${basename} ${ARGN} testUtil) endfunction() ``` -------------------------------- ### C++20 Template Declaration with Concepts Source: https://github.com/ad-freiburg/qlever/wiki/Porting-QLever-to--Cpp17 Demonstrates C++20 template declarations using concepts like std::integral and std::floating_point. ```cpp // C++20, variant a template // `std::integral` and `std::floating_point` are concepts. class C{}; // C++20, variant b template requires (std::integral && std::floating_point) class D{}; ``` -------------------------------- ### Simulating ASK with SELECT and COUNT in SPARQL Source: https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard This workaround simulates the ASK query functionality by using a SELECT query with COUNT to check for the existence of matching patterns. ```sparql SELECT (COUNT(*) > 0 AS ?exists) { ?subject wdt:P31 wd:Q5 . ?subject wdt:P31 wd:Q16521 } ``` -------------------------------- ### Configure CPack Packaging Details Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Sets essential packaging information for CPack, including contact, homepage, output directory, and Debian-specific options. ```cmake set(CPACK_PACKAGE_CONTACT "mundhahj@tf.uni-freiburg.de") set(CPACK_PACKAGE_HOMEPAGE_URL "https://github.com/ad-freiburg/qlever") set(CPACK_OUTPUT_FILE_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/packages") set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS ON) set(CPACK_DEBIAN_FILE_NAME DEB-DEFAULT) set(CPACK_VERBATIM_VARIABLES YES) include(CPack) ``` -------------------------------- ### Add JoinAlgorithmBenchmark Executable Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/CMakeLists.txt Adds and links the 'JoinAlgorithmBenchmark' executable. It depends on 'testUtil' and 'memorySize' libraries in addition to the benchmark infrastructure. ```cmake addAndLinkBenchmark(JoinAlgorithmBenchmark testUtil memorySize) ``` -------------------------------- ### Link Test Binary with Libraries Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Links a specified test binary against Google Mock, the threading library, general test utilities, and any additional specified libraries. ```cmake function(linkTest basename) qlever_target_link_libraries(${basename} ${ARGN} GTest::gtest GTest::gmock_main ${CMAKE_THREAD_LIBS_INIT} global) endfunction() ``` -------------------------------- ### Add Link and Discover Tests (No Libraries) Source: https://github.com/ad-freiburg/qlever/blob/master/test/backports/CMakeLists.txt A macro to add an executable, link it, and discover tests without additional libraries. Applied to multiple test targets. ```cmake addLinkAndDiscoverTestNoLibs(ConceptsTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(ShiftTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(ValueIdentityTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(BackportIteratorTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(StartsWithTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(EndsWithTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(FunctionalTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(ThreeWayComparisonTest) ``` ```cmake addLinkAndDiscoverTestNoLibs(AtomicFlagTest) ``` -------------------------------- ### Apache Jena TDB2 Loader Command Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines This command loads the DBLP dataset into Apache Jena's TDB2 store. It specifies the location for the data and the input file. ```shell apache-jena-5.0.0/bin/tdb2.xloader --loc data dblp.ttl.gz ``` -------------------------------- ### Run ANTLR to Generate Parser Source: https://github.com/ad-freiburg/qlever/blob/master/src/parser/sparqlParser/generated/README.md Use the downloaded ANTLR JAR to generate C++ parser files from the ANTLR grammar definition. Ensure the JAR file and grammar file are in the current directory. ```bash java -jar ./antlr-4.13.2-complete.jar -Dlanguage=Cpp SparqlAutomatic.g4 ``` -------------------------------- ### Define Benchmark Infrastructure Library Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/CMakeLists.txt Compiles the core benchmark infrastructure into a library. This library does not contain a main function and is intended to be linked with other benchmark executables. ```cmake add_library(benchmark infrastructure/Benchmark.cpp infrastructure/BenchmarkMeasurementContainer.cpp infrastructure/BenchmarkToJson.cpp infrastructure/BenchmarkToString.cpp) qlever_target_link_libraries(benchmark Boost::program_options absl::str_format parser util configManager) ``` -------------------------------- ### Process ClueWeb Data with Bash and Python Source: https://github.com/ad-freiburg/qlever/blob/master/misc/clueweb_bash_for_loop.txt This script iterates through files, extracts numerical IDs, calculates context IDs, and processes sentences using a Python script. It runs each iteration in the background. ```bash for f in cw_split_*; do (NUM=$(echo $f | cut -d'_' -f 3); NUM=$((10#$NUM * 1000000)); python3 /local/scratch/bjoern/work/QLever/misc/clueweb_sentences_to_context_and_orig_files.py --stop-tokens-file ~/broccoli/semantic-wikipedia.stop-words --sentences $f --w $f.wordsfile --d $f.docsfile --first-context-id $NUM;) & done ``` -------------------------------- ### Determine Distribution Version Suffix Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt This snippet reads the /etc/os-release file to determine the operating system ID, version ID, and codename. It then constructs a distribution-specific version suffix for packaging. ```cmake set(DEFAULT_VERSION_SUFFIX "") if (EXISTS "/etc/os-release") file(STRINGS "/etc/os-release" OS_RELEASE_CONTENTS) set(_OS_ID "") set(_OS_VERSION_ID "") set(_OS_VERSION_CODENAME "") foreach(line ${OS_RELEASE_CONTENTS}) if (line MATCHES "^ID=\"?([^\"]*)\"?$") set(_OS_ID "${CMAKE_MATCH_1}") elseif (line MATCHES "^VERSION_ID=\"?([^\"]*)\"?$") set(_OS_VERSION_ID "${CMAKE_MATCH_1}") elseif (line MATCHES "^VERSION_CODENAME=\"?([^\"]*)\"?$") set(_OS_VERSION_CODENAME "${CMAKE_MATCH_1}") endif() endforeach() if (_OS_VERSION_CODENAME AND _OS_VERSION_ID) if (_OS_ID STREQUAL "ubuntu") set(DEFAULT_VERSION_SUFFIX "~${_OS_VERSION_CODENAME}~${_OS_VERSION_ID}") elseif (_OS_ID STREQUAL "debian") set(DEFAULT_VERSION_SUFFIX "~${_OS_VERSION_CODENAME}~deb${_OS_VERSION_ID}") endif() endif() endif() set(VERSION_SUFFIX "${DEFAULT_VERSION_SUFFIX}" CACHE STRING "Distribution-specific version suffix") ``` -------------------------------- ### Download Additional Ontologies for PubChem Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-PubChem Downloads various ontologies to enrich PubChem data with labels for identifiers. This script processes a CSV-like input to download and save ontology files. ```bash cut -d, -f3,4 < $NAME ..."; curl --location --silent --remote-time --output rdf.ontologies/$NAME $URL; done BAO - BioAssay Ontology,bao,http://www.bioassayontology.org/bao/bao_complete.owl,bao.rdf BFO - Basic Formal Ontology,bfo,http://purl.obolibrary.org/obo/bfo.owl,bfo.rdf BioPAX - biological pathway data,bp,http://www.biopax.org/release/biopax-level3.owl,bio-pax.rdf CHEMINF - Chemical Information Ontology,cheminf,http://purl.obolibrary.org/obo/cheminf.owl,cheminf.rdf ChEBI - Chemical Entities of Biological Interest,chebi,http://purl.obolibrary.org/obo/chebi.owl,chebi.rdf CiTO,cito,http://purl.org/spar/cito.nt,cito.nt DCMI Terms,dcterms,https://www.dublincore.org/specifications/dublin-core/dcmi-terms/dublin_core_terms.nt,dcterms.nt FaBiO,fabio,http://purl.org/spar/fabio.nt,fabio.nt GO - Gene Ontology,go,http://purl.obolibrary.org/obo/go.owl,go.rdf IAO - Information Artifact Ontology,iao,http://purl.obolibrary.org/obo/iao.owl,iao.rdf NCIt,ncit,http://purl.obolibrary.org/obo/ncit.owl,ncit.rdf NDF-RT,ndfrt,https://data.bioontology.org/ontologies/NDF-RT/submissions/1/download?apikey=8b5b7825-538d-40e0-9e9e-5ab9274a9aeb,ndfrt.rdf OBI - Ontology for Biomedical Investigations,obi,http://purl.obolibrary.org/obo/obi.owl,obi.rdf OWL,owl,http://www.w3.org/2002/07/owl,owl.ttl PDBo,pdbo,http://rdf.wwpdb.org/schema/pdbx-v40.owl,pdbo.rdf PR - PRotein Ontology (PRO),pr,http://purl.obolibrary.org/obo/pr.owl,pr.rdf RDF Schema,rdfs,https://www.w3.org/2000/01/rdf-schema,rdf-schema.ttl,rdfs.ttl RDF,rdf,http://www.w3.org/1999/02/22-rdf-syntax-ns,22-rdf-syntax-ns.ttl,rdf.ttl RO - Relation Ontology,ro,http://purl.obolibrary.org/obo/ro.owl,ro.rdf SIO - Semanticscience Integrated Ontology,sio,http://semanticscience.org/ontology/sio.owl,sio.rdf SKOS,skos,http://www.w3.org/TR/skos-reference/skos.rdf,skos.rdf SO - Sequence types and features ontology,so,http://purl.obolibrary.org/obo/so.owl,so.rdf UO - Units of measurement ontology,uo,http://purl.obolibrary.org/obo/uo.owl,uo.rdf EOT ``` -------------------------------- ### Download UniProt RDF Data Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-UniProt Downloads UniProt RDF data URLs and then downloads the files using wget. Logs download progress and errors. ```bash DATE=2022-10-12 curl -s https://ftp.expasy.org/databases/uniprot/current_release/rdf/RELEASE.meta4 \ | sed 's//' \ | xmllint --xpath '/metalink/files/file/url[@location="ch"]/text()' - \ > uniprot.download-urls.${DATE} mkdir -p rdf.${DATE} > uniprot.${DATE}.download-log cat uniprot.download-urls.${DATE} \ | while read URL; do wget --no-verbose -P rdf.${DATE} ${URL} 2>&1 | tee -a uniprot.${DATE}.download-log; done ``` -------------------------------- ### Make External Dependencies Available Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Makes several external dependencies available for use in the project. This command ensures that libraries declared with FetchContent are ready to be linked. ```cmake FetchContent_MakeAvailable(googletest ctre abseil re2 fsst s2 nlohmann-json antlr range-v3 spatialjoin) ``` -------------------------------- ### Exception Message with Log for Negative Input Source: https://github.com/ad-freiburg/qlever/wiki/Cpp--Style-Guidelines Demonstrates how to log a warning with the problematic value and throw a specific exception for invalid input. Exception messages should include the faulty value and end with a period, while log messages do not. ```cpp // Compute the square root of `x`. Throws `std::out_of_range` on negative `x`. double sqrt(double x) { if (x < 0) { // The messages includes the faulty value of `x`. // No full stop after log. LOG(WARN) << "Trying to take the square root of negative number " << x << std::endl; throw std::out_of_range(absl::StrCat("Cannot compute the square root of negative value ", x, ".")); } return std::sqrt(x); } ``` -------------------------------- ### Download ANTLR Parser Generator Source: https://github.com/ad-freiburg/qlever/blob/master/src/parser/sparqlParser/generated/README.md Download the ANTLR parser generator JAR file. This is a prerequisite for regenerating the parser files. ```bash wget http://www.antlr.org/download/antlr-4.13.2-complete.jar ``` -------------------------------- ### Enable Expensive Unit Tests Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Defines the `QLEVER_RUN_EXPENSIVE_TESTS` preprocessor macro if `RUN_EXPENSIVE_TESTS` is set. A status message indicates this is recommended only in release builds. ```cmake if (RUN_EXPENSIVE_TESTS) message(STATUS "Running expensive unit tests. This is only recommended in release builds") add_compile_definitions(QLEVER_RUN_EXPENSIVE_TESTS) endif () ``` -------------------------------- ### Define sparqlParser Library Source: https://github.com/ad-freiburg/qlever/blob/master/src/parser/sparqlParser/CMakeLists.txt Defines the sparqlParser library and lists its source files, including those generated by ANTLR. ```cmake add_library(sparqlParser generated/SparqlAutomaticBaseListener.cpp generated/SparqlAutomaticLexer.cpp generated/SparqlAutomaticListener.cpp generated/SparqlAutomaticParser.cpp generated/SparqlAutomaticVisitor.cpp) ``` -------------------------------- ### Adding Benchmark Class to CMakeLists.txt Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/Usage.md To compile your benchmark class, add the `addAndLinkBenchmark` command to the `benchmark/CMakeLists.txt` file, specifying the benchmark class file name without the `.cpp` extension. ```cmake addAndLinkBenchmark(MyBenchmarkClassFile) ``` -------------------------------- ### Generate C++ Parser and Lexer Source: https://github.com/ad-freiburg/qlever/blob/master/src/util/ConfigManager/generated/README.md Generates new parser and lexer files for C++ from the ConfigShorthand.g4 grammar file. The -no-listener flag prevents the generation of a default listener, and -Dlanguage=Cpp specifies the target language. ```bash antlr4 -no-listener -Dlanguage=Cpp ConfigShorthand.g4 ``` -------------------------------- ### Add Executable for Algorithm Backport Tests Source: https://github.com/ad-freiburg/qlever/blob/master/test/backports/CMakeLists.txt Defines an executable target named 'AlgorithmBackportTests' and specifies its source file. ```cmake add_executable(AlgorithmBackportTests algorithmTest.cpp) ``` -------------------------------- ### QLever Benchmark Results Table Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-PubChem Displays the performance results for QLever across 16 PubChem use cases, including execution time and number of result rows. ```text Case 1: Protein targets inhibited by ... 1.84 s 17 Case 2: Pharmacological roles of SID4... 0.04 s 13 Case 3: NSAID compounds with molecula... 0.09 s 11 Case 4: NSAID substances according to... 0.24 s 13,867 Case 5: Protein targets inhibited by ... 5.00 s 71 Case 6: Substances inhibiting targets... 3.87 s 149 Case 7: Protein targets inhibited by ... 1.84 s 22 Case 8: Substances inhibiting protein... 1.98 s 9,981 Case 9: Pharmacological roles for sub... 3.83 s 149 Case 10: For each protein, the number... 5.95 s 10,183 Case 11: Top five diseases commonly m... 0.03 s 5 Case 12: Three most recent references... 0.33 s 3 Case 13: Top 20 genes co-mentioned wi... 0.03 s 20 Case 14: Top ten diseases co-occurrin... 0.03 s 10 Case 15: Chemicals commonly mentioned... 0.03 s 1,000 Case 16: Chemicals co-mentioned with ... 0.06 s 274 TOTAL for 16 queries 25.20 s 35,775 AVERAGE for 16 queries 1.57 s 2,236 ``` -------------------------------- ### Link SPARQL Function to C++ Implementation Source: https://github.com/ad-freiburg/qlever/wiki/Implementing-missing-SPARQL-functions-in-QLever Adds an else-if condition to the visitBuiltInCall method in SparqlQleverVisitor.h to map the SPARQL function 'sqr' to the C++ implementation SquareExpression. ```cpp else if (ad_utility::getLowercase(context->children[0]->getText()) == "sqr") { auto child = std::move(visitExpression(context->expression()[0]).as()); return createExpression(std::move(child)); } ``` -------------------------------- ### Link Vocabulary Library Dependencies Source: https://github.com/ad-freiburg/qlever/blob/master/src/index/vocabulary/CMakeLists.txt Links the vocabulary library against other Qlever targets like 'util' and 'rdfTypes'. This ensures that the vocabulary library can use functionality from these dependencies. ```cmake qlever_target_link_libraries(vocabulary util rdfTypes) ``` -------------------------------- ### Fetch GoogleTest Dependency Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Declares the FetchContent module to download the GoogleTest framework from its GitHub repository. It specifies a particular commit hash for reproducibility. ```cmake FetchContent_Declare( googletest GIT_REPOSITORY https://github.com/google/googletest.git GIT_TAG 7917641ff965959afae189afb5f052524395525c # main branch on 2025/09/11 SYSTEM EXCLUDE_FROM_ALL ) ``` -------------------------------- ### Add Subdirectories to Build Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Includes subdirectories into the build process, organizing the project structure. ```cmake add_subdirectory(engine) add_subdirectory(parser) add_subdirectory(index) add_subdirectory(backports) add_subdirectory(rdfTypes) add_subdirectory(joinAlgorithms) add_subdirectory(libqlever) ``` -------------------------------- ### Add and Link Sort Performance Estimator Test Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Compiles and links the SortPerformanceEstimatorTest, but it is commented out by default due to long run time and potential spurious failures. ```cmake # this test runs for quite some time and might have spurious failures! # Therefore it is compiled, but not run. If you want to run it, # change the following two lines. addAndLinkTest(SortPerformanceEstimatorTest SortPerformanceEstimator) #addLinkAndDiscoverTest(SortPerformanceEstimatorTest SortPerformanceEstimator) ``` -------------------------------- ### Include Project Source Directory Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Adds the 'src' directory to the compiler's include paths. This makes it easier to include headers from the project's source files. ```cmake include_directories(src) ``` -------------------------------- ### Convert RDF to Compressed Turtle Source: https://github.com/ad-freiburg/qlever/wiki/Using-QLever-for-UniProt Converts RDF files to compressed Turtle format using Apache Jena and GNU parallel. This process can take over a day. ```bash XML2TTL="apache-jena-3.17.0/bin/rdfxml --output=ttl 2> /dev/null" mkdir -p ttl.${DATE} > rdf2ttl.commands.txt for RDF in rdf.${DATE}/*.{owl,owl.xz,rdf,rdf.xz}; do \ echo "xzcat -f ${RDF} | ${XML2TTL} | xz -c > ttl.${DATE}/$(basename ${RDF} | sed 's/\(rdf\|rdf.xz\|owl\|owl.xz\)$/ttl.xz/') && echo 'DONE converting ${RDF}'" >> rdf2ttl.commands.txt; done cat rdf2ttl.commands | parallel ``` -------------------------------- ### Find First 13 US States by Founding Date Source: https://github.com/ad-freiburg/qlever/blob/master/misc/query-sets/semsearch-no-prefix-queries.fb.txt Retrieves the first 13 US states ordered by their founding date. ```sparql PREFIX fb: SELECT ?x ?val_d WHERE { ?x fb:type.object.type fb:location.us_state . ?x fb:location.dated_location.date_founded ?val_d } ORDER BY ?val_d LIMIT 13 ``` -------------------------------- ### Define Test with Standard Discovery Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Adds a test and links it with standard discovery, suitable for most test cases. ```cmake function(addLinkAndDiscoverTest basename) addTest(${basename}) linkAndDiscoverTest(${basename} ${ARGN}) endfunction() ``` -------------------------------- ### Define Vocabulary Library Source: https://github.com/ad-freiburg/qlever/blob/master/src/index/vocabulary/CMakeLists.txt Defines the vocabulary library and lists its source files. This is a standard CMake command for creating libraries. ```cmake add_library(vocabulary VocabularyInMemory.h VocabularyInMemory.cpp VocabularyInMemoryBinSearch.cpp VocabularyInternalExternal.cpp VocabularyOnDisk.cpp SplitVocabulary.cpp GeoVocabulary.cpp PolymorphicVocabulary.cpp ) ``` -------------------------------- ### QLever Querying Apache Jena Endpoint Source: https://github.com/ad-freiburg/qlever/wiki/QLever-performance-evaluation-and-comparison-to-other-SPARQL-engines This command uses QLever to query the Apache Jena Fuseki SPARQL endpoint. It specifies how to retrieve queries and the endpoint URL. ```shell qlever example-queries --get-queries-cmd "cat queries.tsv" --download-or-count download --sparql-endpoint localhost:8015/dblp ``` -------------------------------- ### Simulating DESCRIBE with SELECT in SPARQL Source: https://github.com/ad-freiburg/qlever/wiki/Current-deviations-from-the-SPARQL-1.1-standard Use this workaround when the DESCRIBE keyword is not supported. It simulates the behavior of DESCRIBE by selecting all triples related to a given entity. ```sparql SELECT ?s ?p ?o WHERE { BIND(wd:Q42 AS ?s) ?s ?p ?o } ``` -------------------------------- ### Define qLever Library Source: https://github.com/ad-freiburg/qlever/blob/master/src/libqlever/CMakeLists.txt Adds the qLever library to the CMake build system, specifying its source file. ```cmake add_library(qlever Qlever.cpp) ``` -------------------------------- ### Link test utility library Source: https://github.com/ad-freiburg/qlever/blob/master/test/util/CMakeLists.txt Links the 'testUtil' library against other targets such as 'engine', 'global', and 'gmock_main'. This ensures that the test utility has access to the functionality provided by these dependencies during compilation and linking. ```cmake qlever_target_link_libraries(testUtil engine global gmock_main) ``` -------------------------------- ### Colored Output for Ninja Generator Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Enables colored diagnostics for the Ninja build system generator. This improves readability of compiler output by color-coding messages. ```cmake if ("${CMAKE_GENERATOR}" STREQUAL "Ninja") if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU") add_compile_options(-fdiagnostics-color=always) elseif ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang" OR "${CMAKE_CXX_COMPILER_ID}" STREQUAL "AppleClang") add_compile_options(-fcolor-diagnostics) endif () ``` -------------------------------- ### Registering a Benchmark Class with AD_REGISTER_BENCHMARK Source: https://github.com/ad-freiburg/qlever/blob/master/benchmark/Usage.md This macro is used to register your custom benchmark class with the ad-benchmark framework. It takes the class name and any constructor arguments as parameters. Ensure this is called within the ad_benchmark namespace. ```c++ AD_REGISTER_BENCHMARK(MyClass, ConstructorArgument1, ConstructorArgument2, ...); ``` -------------------------------- ### Add Test for Query Planner Source: https://github.com/ad-freiburg/qlever/blob/master/test/CMakeLists.txt Adds tests for QueryPlanner and QueryPlannerSpatialJoin, requiring serial execution due to potential file conflicts. ```cmake addLinkAndDiscoverTestSerial(QueryPlannerTest engine) ``` ```cmake addLinkAndDiscoverTestSerial(QueryPlannerSpatialJoinTest engine) ``` -------------------------------- ### Display CMake Build Information Source: https://github.com/ad-freiburg/qlever/blob/master/CMakeLists.txt Prints the current values of CMAKE_CXX_FLAGS, CMAKE_CXX_FLAGS_RELEASE, CMAKE_CXX_FLAGS_DEBUG, and CMAKE_BUILD_TYPE to the console. This is useful for debugging build configurations. ```cmake message(STATUS ---) message(STATUS "CXX_FLAGS are : " ${CMAKE_CXX_FLAGS}) message(STATUS "CXX_FLAGS_RELEASE are : " ${CMAKE_CXX_FLAGS_RELEASE}) message(STATUS "CXX_FLAGS_DEBUG are : " ${CMAKE_CXX_FLAGS_DEBUG}) message(STATUS "IMPORTANT: Make sure you have selected the desired CMAKE_BUILD_TYPE") message(STATUS "CMAKE_BUILD_TYPE is ${CMAKE_BUILD_TYPE}") message(STATUS ---) ```