### Install Elastic Client v2 for Elasticsearch 1.x Source: https://github.com/olivere/elastic/wiki/Changes Installs version 2.0 of the Elastic client library, which is compatible with Elasticsearch versions 1.x. This command uses the `go get` utility to fetch and install the specified package. ```sh go get gopkg.in/olivere/elastic.v2 ``` -------------------------------- ### Start and Configure Bulk Processor Source: https://github.com/olivere/elastic/wiki/BulkProcessor Starts a configured bulk processor by calling the `Do` method on the builder. This method spins up goroutines to handle bulk requests and accepts a context for long-running operations. ```go client, err := elastic.NewClient() if err != nil { ... } // Setup a bulk processor p, err := client.BulkProcessor().Name("MyBackgroundWorker-1").Workers(2).Do(context.Background()) if err != nil { ... } ``` -------------------------------- ### Setup Basic Bulk Processor Source: https://github.com/olivere/elastic/wiki/BulkProcessor Initializes a new Elasticsearch client and sets up a basic bulk processor with a specified name. This is the first step to creating a background service for bulk requests. ```go client, err := elastic.NewClient() if err != nil { ... } // Setup a bulk processor service := client.BulkProcessor().Name("MyBackgroundWorker-1") ``` -------------------------------- ### Example Cluster Test Execution Source: https://github.com/olivere/elastic/blob/release-branch.v7/cluster-test/README.md An example command demonstrating how to run the cluster-test program with specific configurations for testing an Elasticsearch cluster. It specifies node URLs, number of search goroutines, index name, retries, and health/sniffer settings. ```sh $ ./cluster-test -nodes=http://127.0.0.1:9200,http://127.0.0.1:9201,http://127.0.0.1:9202 -n=5 -index=twitter -retries=5 -sniff=true -sniffer=10s -healthcheck=true -healthchecker=5s -errorlog=error.log ``` -------------------------------- ### Create Elastic Client for Elasticsearch 1.x (Go) Source: https://github.com/olivere/elastic/wiki/Connection-Problems This Go code example demonstrates creating an Elastic client for Elasticsearch 1.x. It uses the `gopkg.in/olivere/elastic.v2` import path and includes a check for errors returned by `elastic.NewClient()`. ```go import ( "net/http" "gopkg.in/olivere/elastic.v2" ) // Create a client for Elasticsearch 1.x client, err := elastic.NewClient() if err != nil { // Handle error panic(err) } ``` -------------------------------- ### Example of Using DeleteByQueryService with Elastic 3.0 (Plugin Required) Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-3.0.md Demonstrates how to use the DeleteByQueryService in Elastic 3.0. Note that this requires the 'delete-by-query' plugin to be installed; otherwise, it will return an ErrPluginNotFound error. This example shows a common pattern for attempting the operation and handling the specific plugin not found error. ```go // Example for Elastic 3.0 (new): // _, err := client.DeleteByQuery().Query(elastic.NewTermQuery("client", "1")).Do() // if err == elastic.ErrPluginNotFound { // // Delete By Query API is not available // } ``` -------------------------------- ### Create Elastic Client for Elasticsearch 5.x (Go) Source: https://github.com/olivere/elastic/wiki/Connection-Problems This code example illustrates creating an Elastic client for Elasticsearch version 5.x. It uses the `gopkg.in/olivere/elastic.v5` import path and includes basic error handling for the client creation process. ```go import ( "net/http" "gopkg.in/olivere/elastic.v5" ) // Notice the v5 here! // Create a client for Elasticsearch 5.x client, err := elastic.NewClient() if err != nil { // Handle error panic(err) } ``` -------------------------------- ### Elasticsearch Film Finder Example in Go Source: https://github.com/olivere/elastic/wiki/QueryDSL This Go program demonstrates a comprehensive film finder using the olivere/elastic library. It sets up an Elasticsearch index for films, defines a 'Finder' struct with methods for filtering and pagination, and executes search queries. The output includes total films found, a list of films, and breakdowns by genre and year. Key dependencies include 'context', 'encoding/json', 'fmt', 'time', and 'github.com/olivere/elastic'. ```go package main import ( "context" "encoding/json" "fmt" _ "log" "strings" "time" "github.com/olivere/elastic" ) const ( indexName = "films" mapping = ` { "settings":{ "number_of_shards":1, "number_of_replicas":0 }, "mappings":{ "_doc":{ "properties":{ "title":{ "type":"keyword" }, "genre":{ "type":"keyword" }, "year":{ "type":"long" }, "director":{ "type":"keyword" } } } } } ` ) func main() { opts := []elastic.ClientOptionFunc{ // Uncomment next line to show response from Elasticsearch //elastic.SetTraceLog(log.New(os.Stdout, "", 0)), } client, err := elastic.NewClient(opts...) if err != nil { panic(err) } // Create some sample films err = createAndPopulateIndex(client) if err != nil { panic(err) } // Create a finder f := NewFinder() // f = f.Year(2014) f = f.From(0).Size(100) f = f.Pretty(true) // Provide a timeout of 5 seconds ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() // Execute the finder res, err := f.Find(ctx, client) if err != nil { panic(err) } // Output results fmt.Printf("Searched through %d films\n", res.Total) fmt.Println() fmt.Println("Films found:") for i, film := range res.Films { prefix := "├" if i == len(res.Films)-1 { prefix = "└" } fmt.Printf("%s %s from %d\n", prefix, film.Title, film.Year) } fmt.Println() fmt.Println("Broken down by genre:") for genre, count := range res.Genres { fmt.Printf("- %2d× %s\n", count, genre) } fmt.Println() fmt.Println("Broken down by year and genres:") for year, genre := range res.YearsAndGenres { fmt.Printf("- %4d\n", year) for i, nc := range genre { prefix := "├" if i == len(genre)-1 { prefix = "└" } fmt.Printf(" %s%2d× %s\n", prefix, nc.Count, nc.Name) } } } // Film represents a movie with some properties. type Film struct { Title string `json:"title"` Genre []string `json:"genre"` Year int `json:"year"` Director string `json:"director"` } // Finder specifies a finder for films. type Finder struct { genre string year int from, size int sort []string pretty bool } // FinderResponse is the outcome of calling Finder.Find. type FinderResponse struct { Total int64 Films []*Film Genres map[string]int64 YearsAndGenres map[int][]NameCount // {1994: [{"Crime":1}, {"Drama":2}], ...} } // NameCount represents a name associated with a count. type NameCount struct { Name string Count int64 } // NewFinder creates a new finder for films. // Use the funcs to set up filters and search properties, // then call Find to execute. func NewFinder() *Finder { return &Finder{} } // Genre filters the results by the given genre. func (f *Finder) Genre(genre string) *Finder { f.genre = genre return f } // Year filters the results by the specified year. func (f *Finder) Year(year int) *Finder { f.year = year return f } // From specifies the start index for pagination. func (f *Finder) From(from int) *Finder { f.from = from return f } // Size specifies the number of items to return in pagination. func (f *Finder) Size(size int) *Finder { f.size = size return f } // Sort specifies one or more sort orders. ``` -------------------------------- ### Custom Retrier Implementation Example Source: https://github.com/olivere/elastic/wiki/Retrier-and-Backoff Demonstrates how to implement a custom Retrier in Go. This example uses an ExponentialBackoff strategy, includes a specific error check to stop retries immediately, and limits the total number of retries. ```go type MyRetrier struct { backoff elastic.Backoff } func NewMyRetrier() *MyRetrier { return &MyRetrier{ backoff: elastic.NewExponentialBackoff(10 * time.Millisecond, 8 * time.Second), } } func (r *MyRetrier) Retry(ctx context.Context, retry int, req *http.Request, resp *http.Response, err error) (time.Duration, bool, error) { // Fail hard on a specific error if err == syscall.ECONNREFUSED { return 0, false, errors.New("Elasticsearch or network down") } // Stop after 5 retries if retry >= 5 { return 0, false, nil } // Let the backoff strategy decide how long to wait and whether to stop wait, stop := r.backoff.Next(retry) return wait, stop, nil } // Usage example: // client, err := elastic.NewClient( // elastic.SetURL("http://127.0.0.1:9200"), // elastic.SetRetrier(NewMyRetrier()), // ) // if err != nil { ... } ``` -------------------------------- ### Enable All Logging Types (Error, Info, Trace) Source: https://github.com/olivere/elastic/wiki/Logging An example demonstrating how to enable all three types of logs: error, info, and trace. It configures distinct loggers and output streams for each type. ```go client, err := elastic.NewClient( elastic.SetErrorLog(log.New(os.Stderr, "ELASTIC ", log.LstdFlags)), elastic.SetInfoLog(log.New(os.Stdout, "", log.LstdFlags))), elastic.SetTraceLog(log.New(os.Stderr, "[[ELASTIC]]", 0)))) ``` -------------------------------- ### Elasticsearch Cluster Nodes Info API Example Source: https://github.com/olivere/elastic/wiki/Sniffing An example of the output from the Elasticsearch Cluster Nodes Info API when queried for HTTP information. This output is used by the client during the sniffing process to discover cluster nodes and their addresses. ```sh $ curl -XGET 'http://127.0.0.1:9200/_nodes/http?pretty=true' { "cluster_name" : "elasticsearch", "nodes" : { "msEWB6vOQ6qVzC2y8fTOhw" : { "name" : "Blackheath", "transport_address" : "inet[/127.0.0.1:9300]", "host" : "aero", "ip" : "192.168.10.10", "version" : "1.4.4", "build" : "c88f77f", "http_address" : "inet[/127.0.0.1:9200]", "http" : { "bound_address" : "inet[/127.0.0.1:9200]", "publish_address" : "inet[/127.0.0.1:9200]" } } } } ``` -------------------------------- ### Configure Elasticsearch Client Options in Go Source: https://github.com/olivere/elastic/wiki/Configuration Demonstrates how to configure a new Elasticsearch client using various options. This example shows setting multiple URLs, disabling sniffing, adjusting healthcheck intervals, implementing a custom retrier, enabling Gzip compression, configuring error and info logs, and setting custom HTTP headers. It requires the 'log' and 'os' packages for logging and the 'time' package for durations. ```go import ( "log" "net/http" "os" "time" "github.com/olivere/elastic/v7" ) // Assume NewCustomRetrier() is defined elsewhere // func NewCustomRetrier() elastic.Retrier { ... } func main() { // Obtain a client for an Elasticsearch cluster of two nodes, // running on 10.0.1.1 and 10.0.1.2. Do not run the sniffer. // Set the healthcheck interval to 10s. When requests fail, // retry 5 times. Print error messages to os.Stderr and informational // messages to os.Stdout. client, err := elastic.NewClient( elastic.SetURL("http://10.0.1.1:9200", "http://10.0.1.2:9200"), elastic.SetSniff(false), elastic.SetHealthcheckInterval(10*time.Second), elastic.SetRetrier(NewCustomRetrier()), elastic.SetGzip(true), elastic.SetErrorLog(log.New(os.Stderr, "ELASTIC ", log.LstdFlags)), elastic.SetInfoLog(log.New(os.Stdout, "", log.LstdFlags)), elastic.SetHeaders(http.Header{ "X-Caller-Id": []string{"my-app-version-1.0"}, }), ) if err != nil { panic(err) } // Use the client... _ = client } ``` -------------------------------- ### Reindex API Example (Go) Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-5.0.md Demonstrates how to use the native Reindex API via the 'client.Reindex()' method in the olivere/elastic Go client. This replaces the older custom ReindexerService. It takes an Elasticsearch client and a Reindex request object. ```Go // Assuming 'client' is an initialized elastic.Client // and 'reindexRequest' is an elastic.ReindexRequest object response, err := client.Reindex().Do(ctx, reindexRequest) if err != nil { // Handle error } // Process the response ``` -------------------------------- ### Set Required Plugins for Elastic Client Initialization Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-3.0.md This client option ensures that specified plugins are installed before the client is created. If any of the required plugins are missing, client initialization will fail with an error. This proactively prevents runtime errors when using features dependent on these plugins. ```go // Will raise an error if the "delete-by-query" plugin is NOT installed client, err := elastic.NewClient(elastic.SetRequiredPlugins("delete-by-query")) if err != nil { log.Fatal(err) } ``` -------------------------------- ### Elasticsearch Client with AWS SigV4 using aws_signing_client Source: https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service Creates an Elasticsearch client for AWS by integrating with the `aws_signing_client` package, which uses AWS SDK for Go's v4 signer. This example demonstrates setting up credentials and configuring the HTTP client for authenticated requests to AWS Elasticsearch Service. ```go import ( "github.com/aws/aws-sdk-go/aws/signer/v4" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/sha1sum/aws_signing_client" "gopkg.in/olivere/elastic.v3" ) func newElasticClient(creds *credentials.Credentials) (*elastic.Client, error) { signer := v4.NewSigner(creds) awsClient, err := aws_signing_client.New(signer, nil, "es", "us-east-1") if err != nil { return nil, err } return elastic.NewClient( elastic.SetURL("https://my-aws-endpoint.us-east-1.es.amazonaws.com"), elastic.SetScheme("https"), elastic.SetHttpClient(awsClient), elastic.SetSniff(false), // See note below ) } ``` -------------------------------- ### Get, Update, and Delete Documents with Go Source: https://context7.com/olivere/elastic/llms.txt Shows how to retrieve, update, and delete documents in Elasticsearch using the oliveder/elastic Go client. Includes handling of document not found and connection errors, and updating with a script. Requires the elastic/v7 package. ```go package main import ( "context" "encoding/json" "fmt" "log" "github.com/olivere/elastic/v7" ) type Tweet struct { User string `json:"user"` Message string `json:"message"` Retweets int `json:"retweets"` } func main() { client, err := elastic.NewClient() if err != nil { log.Fatal(err) } ctx := context.Background() // Get a document by ID get1, err := client.Get(). Index("twitter"). Id("1"). Do(ctx) if err != nil { switch { case elastic.IsNotFound(err): log.Printf("Document not found: %v", err) case elastic.IsTimeout(err): log.Printf("Timeout retrieving document: %v", err) case elastic.IsConnErr(err): log.Printf("Connection problem: %v", err) default: log.Printf("Error: %v", err) } return } // Deserialize the document var tweet Tweet if err := json.Unmarshal(get1.Source, &tweet); err != nil { log.Fatalf("Error deserializing: %v", err) } fmt.Printf("Got tweet by %s: %s (retweets: %d)\n", tweet.User, tweet.Message, tweet.Retweets) // Update a document using a script script := elastic.NewScript("ctx._source.retweets += params.num"). Param("num", 1) update, err := client.Update(). Index("twitter"). Id("1"). Script(script). Upsert(map[string]interface{}{"retweets": 0}). Do(ctx) if err != nil { log.Fatalf("Error updating document: %v", err) } fmt.Printf("Updated document %s to version %d\n", update.Id, update.Version) // Delete a document delete, err := client.Delete(). Index("twitter"). Id("1"). Do(ctx) if err != nil { log.Fatalf("Error deleting document: %v", err) } fmt.Printf("Deleted document %s from index %s\n", delete.Id, delete.Index) } ``` -------------------------------- ### Configure Bulk Processor with All Options (Go) Source: https://github.com/olivere/elastic/wiki/BulkProcessor Demonstrates setting up a bulk processor with all available configuration options. This includes setting a name, defining before and after callbacks, specifying the number of workers, and configuring thresholds for bulk actions, bulk size, and flush interval. Statistics collection is also enabled. ```go // Setup a bulk processor p, err := client.BulkProcessor(). Name("MyBackgroundWorker-1"). Before(beforeCallback). // func to call before commits After(afterCallback). // func to call after commits Workers(4). // number of workers BulkActions(1000). // commit if # requests >= 1000 BulkSize(2 << 20). // commit if size of requests >= 2 MB FlushInterval(30*time.Second). // commit every 30s Stats(true). // collect stats Do(context.Background()) if err != nil { panic(err) } ``` -------------------------------- ### Initialize Basic Elasticsearch Client Source: https://context7.com/olivere/elastic/llms.txt Connects to a default Elasticsearch instance running at http://127.0.0.1:9200 and verifies the connection by pinging the cluster. It retrieves and prints the Elasticsearch version. Requires the 'context' and 'fmt' packages. ```go package main import ( "context" "fmt" "log" "github.com/olivere/elastic/v7" ) func main() { // Connect to default Elasticsearch instance at http://127.0.0.1:9200 client, err := elastic.NewClient() if err != nil { log.Fatalf("Error creating client: %v", err) } // Verify connection by pinging the cluster info, code, err := client.Ping("http://127.0.0.1:9200").Do(context.Background()) if err != nil { log.Fatalf("Ping failed: %v", err) } fmt.Printf("Elasticsearch version %s (code %d)\n", info.Version.Number, code) // Get Elasticsearch version esversion, err := client.ElasticsearchVersion("http://127.0.0.1:9200") if err != nil { log.Fatalf("Error getting version: %v", err) } fmt.Printf("Connected to Elasticsearch %s\n", esversion) } ``` -------------------------------- ### Cluster Test Program Help Output Source: https://github.com/olivere/elastic/blob/release-branch.v7/cluster-test/README.md Displays the usage instructions and available command-line flags for the cluster-test program. This helps users understand the configuration options for testing their Elasticsearch cluster. ```sh $ ./cluster-test -h Usage of ./cluster-test: -errorlog="": error log file -healthcheck=true: enable or disable healthchecks -healthchecker=1m0s: healthcheck interval -index="twitter": name of ES index to use -infolog="": info log file -n=5: number of goroutines that run searches -nodes="": comma-separated list of ES URLs (e.g. 'http://192.168.2.10:9200,http://192.168.2.11:9200') -retries=0: number of retries -sniff=true: enable or disable sniffer -sniffer=15m0s: sniffer interval -tracelog="": trace log file ``` -------------------------------- ### Check for Delete-by-Query Plugin Installation in Elastic Client Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-3.0.md This helper function checks if the 'delete-by-query' plugin is installed and available for use. It returns a boolean indicating presence and an error if any occurred during the check. Essential for verifying plugin availability before attempting to use plugin-dependent APIs. ```go err, found := client.HasPlugin("delete-by-query") if err == nil && found { // ... Delete By Query API is available } ``` -------------------------------- ### Iterate Large Elasticsearch Results with Go ScrollService and Goroutines Source: https://github.com/olivere/elastic/wiki/ScrollParallel This Go code snippet demonstrates how to efficiently iterate through a large Elasticsearch result set using the ScrollService. It leverages multiple goroutines, managed by `golang.org/x/sync/errgroup`, to concurrently fetch and deserialize documents. The example includes setting up an Elasticsearch client, counting total documents, initializing a progress bar, and processing documents through channels. Dependencies include `golang.org/x/net/context`, `golang.org/x/sync/errgroup`, `gopkg.in/cheggaaa/pb.v1`, and `gopkg.in/olivere/elastic.v3`. The primary input is an Elasticsearch index and type, and the output is processed documents. ```go package main import ( "encoding/json" "io" "golang.org/x/net/context" "golang.org/x/sync/errgroup" "gopkg.in/cheggaaa/pb.v1" "gopkg.in/olivere/elastic.v3" ) type Product struct { SKU string `json:"sku"` Name string `json:"name"` } func main() { client, err := elastic.NewClient() if err != nil { panic(err) } // Count total and setup progress total, err := client.Count("warehouse").Type("product").Do() if err != nil { panic(err) } bar := pb.StartNew(int(total)) // This example illustrates how to use goroutines to iterate // through a result set via ScrollService. // // It uses the excellent golang.org/x/sync/errgroup package to do so. // // The first goroutine will Scroll through the result set and send // individual documents to a channel. // // The second cluster of goroutines will receive documents from the channel and // deserialize them. // // Feel free to add a third goroutine to do something with the // deserialized results. // // Let's go. // 1st goroutine sends individual hits to channel. hits := make(chan json.RawMessage) g, ctx := errgroup.WithContext(context.Background()) g.Go(func() error { defer close(hits) // Initialize scroller. Just don't call Do yet. scroll := client.Scroll("warehouse").Type("product").Size(100) for { results, err := scroll.Do() if err == io.EOF { return nil // all results retrieved } if err != nil { return err // something went wrong } // Send the hits to the hits channel for _, hit := range results.Hits.Hits { select { case hits <- *hit.Source: case <-ctx.Done(): return ctx.Err() } } } return nil }) // 2nd goroutine receives hits and deserializes them. // // If you want, setup a number of goroutines handling deserialization in parallel. for i := 0; i < 10; i++ { g.Go(func() error { for hit := range hits { // Deserialize var p Product err := json.Unmarshal(hit, &p) if err != nil { return err } // Do something with the product here, e.g. send it to another channel // for further processing. _ = p bar.Increment() // Terminate early? select { default: case <-ctx.Done(): return ctx.Err() } } return nil }) } // Check whether any goroutines failed. if err := g.Wait(); err != nil { panic(err) } // Done. bar.FinishPrint("Done") } ``` -------------------------------- ### Percolator Query Example (JSON) Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-5.0.md Demonstrates the use of the new Percolator Query in Elasticsearch 5.x, which can be utilized with the Percolator type. This signifies a significant change from the previous Percolate service. ```JSON { "query": { "percolate": { "field": "pin.location", "document_type": "pin", "index": "my_index", "document": { "location": { "lat": 40.7128, "lon": -74.0060 } } } } } ``` -------------------------------- ### PerformRequest and Do Context Enforcement Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-5.0.md Starting with Elasticsearch 5.0, the usage of `context.Context` is enforced in `PerformRequest` and `Do` methods for automatic request cancelation and other context-aware patterns. Users must update their `Do()` calls to `Do(ctx)`. ```APIDOC ## Context Enforcement in Request Execution ### Description This change enforces the use of `context.Context` in all request execution methods, such as `PerformRequest` and `Do`. This allows for better control over request lifecycles, including cancellation. ### Method Applies to methods like `Do()`. ### Endpoint N/A (Code-level change) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```go // Before 5.0 err := client.Do(req) // After 5.0 ctx := context.Background() // or context.TODO() err := client.Do(ctx, req) ``` ### Response N/A (Code-level change) #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### Update API (v3 and above) Source: https://github.com/olivere/elastic/wiki/Update This section demonstrates how to update a document using the Update API in Elasticsearch versions 3 and above. It includes an example of incrementing a field using a script and providing parameters. ```APIDOC ## POST ///_update/ ### Description Updates a document at the specified index, type, and ID using a script. This is the recommended approach for versions 3 and above. ### Method POST ### Endpoint `///_update/` ### Parameters #### Path Parameters - **index** (string) - Required - The name of the index. - **type** (string) - Required - The type of the document. - **id** (string) - Required - The ID of the document to update. #### Query Parameters None specified in the example. #### Request Body - **script** (object) - Required - Defines the script to execute for the update. - **source** (string) - The script source code (e.g., `ctx._source.retweets += params.num`). - **params** (object) - Parameters to be used within the script. - **num** (integer) - Example parameter for the script. - **upsert** (object) - Optional - The document to insert if it does not exist. - **retweets** (integer) - Example field for the upsert document. ### Request Example ```go ctx := context.Background() update, err := client.Update().Index("twitter").Type("tweet").Id("1"). Script(elastic.NewScript("ctx._source.retweets += params.num").Param("num", 1)). Upsert(map[string]interface{}{"retweets": 0}). Do(ctx) if err != nil { // Handle error panic(err) } fmt.Printf("New version of tweet %q is now %d", update.Id, update.Version) ``` ### Response #### Success Response (200) - **_id** (string) - The ID of the updated document. - **_version** (integer) - The new version of the document. #### Response Example (Output would typically be Elasticsearch JSON response, not shown in Go code example) ``` -------------------------------- ### Continuous Bulk Indexing with BulkProcessor in Go Source: https://context7.com/olivere/elastic/llms.txt Shows how to set up and use a `BulkProcessor` for continuous indexing in Elasticsearch. This approach allows for configurable batch sizes, flush intervals, and custom callbacks for pre- and post-bulk operations. It requires an Elasticsearch client to be initialized. ```go package main import ( "context" "fmt" "log" "time" "github.com/google/uuid" "github.com/olivere/elastic/v7" ) type Document struct { Timestamp time.Time `json:"@timestamp"` Message string `json:"message"` } func main() { client, err := elastic.NewClient() if err != nil { log.Fatal(err) } ctx := context.Background() // Create bulk processor with custom settings p, err := client.BulkProcessor(). Name("my-bulk-processor"). Workers(4). BulkActions(1000). BulkSize(2 << 20). // 2 MB FlushInterval(5 * time.Second). Stats(true). Before(func(executionId int64, requests []elastic.BulkableRequest) { fmt.Printf("About to commit %d requests\n", len(requests)) }). After(func(executionId int64, requests []elastic.BulkableRequest, response *elastic.BulkResponse, err error) { if err != nil { log.Printf("Bulk commit failed: %v", err) } else { fmt.Printf("Committed %d requests\n", len(requests)) } }). Do(ctx) if err != nil { log.Fatal(err) } defer p.Close() // Add documents to bulk processor for i := 0; i < 10000; i++ { doc := Document{ Timestamp: time.Now(), Message: fmt.Sprintf("Document %d", i), } req := elastic.NewBulkIndexRequest(). Index("logs"). Id(uuid.New().String()). Doc(doc) p.Add(req) } // Flush remaining documents if err := p.Flush(); err != nil { log.Fatal(err) } // Get statistics stats := p.Stats() fmt.Printf("Succeeded: %d, Failed: %d, Committed: %d\n", stats.Succeeded, stats.Failed, stats.Committed) } ``` -------------------------------- ### Initialize Elasticsearch Client with Custom Configuration Source: https://context7.com/olivere/elastic/llms.txt Creates an Elasticsearch client with advanced configuration options, including multiple node URLs, disabling sniffing, setting health check intervals, defining maximum retries, and configuring custom error and info loggers. It also enables gzip compression. Requires 'log', 'os', and 'time' packages. ```go package main import ( "log" "os" "time" "github.com/olivere/elastic/v7" ) func main() { // Configure client with multiple nodes, custom timeouts, and logging client, err := elastic.NewClient( elastic.SetURL("http://10.0.1.1:9200", "http://10.0.1.2:9200"), elastic.SetSniff(false), elastic.SetHealthcheckInterval(10*time.Second), elastic.SetMaxRetries(5), elastic.SetErrorLog(log.New(os.Stderr, "ELASTIC ", log.LstdFlags)), elastic.SetInfoLog(log.New(os.Stdout, "INFO ", log.LstdFlags)), elastic.SetGzip(true), ) if err != nil { log.Fatalf("Failed to create client: %v", err) } log.Printf("Client created successfully with %d nodes", 2) _ = client } ``` -------------------------------- ### Configure Client with go-kit/kit Logger Source: https://github.com/olivere/elastic/wiki/Logging Illustrates how to integrate the Elasticsearch client with the go-kit/kit logging library. A wrapper struct 'wrapKitLogger' is provided to implement the required 'Printf' method. ```go import "github.com/go-kit/kit/log" type wrapKitLogger struct { log.Logger } func (logger wrapKitLogger) Printf(format string, vars ...interface{}) { logger.Log("msg", fmt.Sprintf(format, vars...)) } ... logger := log.NewLogfmtLogger(os.Stdout) wrappedLogger := &wrapKitLogger{logger} client, err := elastic.NewClient(elastic.SetErrorLog(wrappedLogger)) ``` -------------------------------- ### Basic Search with Term Query using Go Source: https://context7.com/olivere/elastic/llms.txt Demonstrates performing a basic search in Elasticsearch using a term query on the 'user' field with the oliveder/elastic Go client. It shows how to execute the query, sort results, and iterate through the found documents. Requires the elastic/v7 package. ```go package main import ( "context" "encoding/json" "fmt" "log" "github.com/olivere/elastic/v7" ) type Tweet struct { User string `json:"user"` Message string `json:"message"` } func main() { client, err := elastic.NewClient() if err != nil { log.Fatal(err) } ctx := context.Background() // Create a term query termQuery := elastic.NewTermQuery("user", "olivere") // Execute search searchResult, err := client.Search(). Index("twitter"). Query(termQuery). Sort("user", true). From(0).Size(10). Pretty(true). Do(ctx) if err != nil { log.Fatalf("Error searching: %v", err) } fmt.Printf("Query took %d milliseconds\n", searchResult.TookInMillis) fmt.Printf("Found %d documents\n", searchResult.TotalHits()) // Iterate through results if searchResult.TotalHits() > 0 { for _, hit := range searchResult.Hits.Hits { var tweet Tweet if err := json.Unmarshal(hit.Source, &tweet); err != nil { log.Printf("Error deserializing: %v", err) continue } fmt.Printf("Tweet by %s: %s (score: %f)\n", tweet.User, tweet.Message, *hit.Score) } } } ``` -------------------------------- ### Chaos Monkey Script for Elasticsearch Node Simulation Source: https://github.com/olivere/elastic/blob/release-branch.v7/cluster-test/README.md A bash script designed to simulate Elasticsearch nodes going up and down. It repeatedly starts and stops Elasticsearch instances with random sleep intervals, useful for testing cluster resilience. ```bash #!/bin/bash while true do echo "Starting ES node" elasticsearch -d -Xmx4g -Xms1g -Des.config=elasticsearch.yml -p es.pid sleep `jot -r 1 10 300` # wait for 10-300s echo "Stopping ES node" kill -TERM `cat es.pid` sleep `jot -r 1 10 60` # wait for 10-60s done ``` -------------------------------- ### Create a Simple Elasticsearch Client Source: https://github.com/olivere/elastic/wiki/Client Illustrates how to create a lightweight, simple Elasticsearch client. This client disables advanced features like sniffing and health checks, making it suitable for restricted environments. Error handling and client stopping are also included. ```go // Create a simple client client, err := elastic.NewSimpleClient() if err != nil { // Handle error panic(err) } def client.Stop() ... ``` -------------------------------- ### Bulk Indexing with BulkService in Go Source: https://context7.com/olivere/elastic/llms.txt Demonstrates bulk indexing of documents using the `BulkService` in the Olivere Go client. It uses goroutines for producing documents and consuming them in bulk. Requires an Elasticsearch instance running on localhost:9200. ```go package main import ( "context" "encoding/base64" "fmt" "log" "math/rand" "time" "github.com/olivere/elastic/v7" "golang.org/x/sync/errgroup" ) type Doc struct { ID string `json:"id"` Timestamp time.Time `json:"@timestamp"` } func main() { client, err := elastic.NewClient( elastic.SetURL("http://localhost:9200"), elastic.SetSniff(false), ) if err != nil { log.Fatal(err) } ctx := context.Background() g, ctx := errgroup.WithContext(ctx) // Channel for documents docsc := make(chan Doc, 100) numDocs := 10000 bulkSize := 1000 // Producer goroutine g.Go(func() error { defer close(docsc) buf := make([]byte, 32) for i := 0; i < numDocs; i++ { rand.Read(buf) id := base64.URLEncoding.EncodeToString(buf) docsc <- Doc{ID: id, Timestamp: time.Now()} } return nil }) // Consumer goroutine - bulk insert g.Go(func() error { bulk := client.Bulk().Index("documents") for d := range docsc { bulk.Add(elastic.NewBulkIndexRequest().Id(d.ID).Doc(d)) if bulk.NumberOfActions() >= bulkSize { res, err := bulk.Do(ctx) if err != nil { return fmt.Errorf("bulk commit failed: %w", err) } if res.Errors { for _, item := range res.Failed() { log.Printf("Failed item: %v", item.Error) } return fmt.Errorf("bulk had errors") } fmt.Printf("Committed %d documents\n", bulkSize) } } // Commit remaining documents if bulk.NumberOfActions() > 0 { _, err := bulk.Do(ctx) return err } return nil }) if err := g.Wait(); err != nil { log.Fatal(err) } fmt.Println("Bulk indexing completed") } ``` -------------------------------- ### Enforce context.Context in PerformRequest and Do Source: https://github.com/olivere/elastic/blob/release-branch.v7/CHANGELOG-5.0.md Starting with Elastic 5.0, context.Context is enforced for request execution. All `Do()` calls must now accept a context argument, enabling automatic request cancellation and other patterns. Use `context.TODO()` or `context.Background()` if explicit context is not required. ```go import ( "context" "github.com/olivere/elastic/v7" ) func main() { client, err := elastic.NewClient() if err != nil { panic(err) } // Example of passing a context ctx := context.Background() _, err = client.PerformRequest(ctx, "GET", "/", nil) if err != nil { panic(err) } // Example of passing context.TODO() _, err = client.Do(context.TODO()) if err != nil { panic(err) } } ``` -------------------------------- ### Create Elastic Client for Elasticsearch 2.x (Go) Source: https://github.com/olivere/elastic/wiki/Connection-Problems This snippet provides the Go code to initialize an Elastic client for Elasticsearch 2.x, utilizing the `gopkg.in/olivere/elastic.v3` import path. It includes a standard error check after calling `elastic.NewClient()`. ```go import ( "net/http" "gopkg.in/olivere/elastic.v3" ) // Notice the v3 here! // Create a client for Elasticsearch 2.x client, err := elastic.NewClient() if err != nil { // Handle error panic(err) } ``` -------------------------------- ### Create Elasticsearch Client with Multiple URLs Source: https://github.com/olivere/elastic/wiki/Client Shows how to initialize an Elasticsearch client with a list of URLs, allowing connection to multiple nodes in a cluster. The library automatically handles node discovery. Error handling and client stopping are included. ```go // Create a client and connect to nodes http://127.0.0.1:9200 and http://127.0.0.1:9201 client, err := elastic.NewClient(elastic.SetURL("http://127.0.0.1:9200", "http://127.0.0.1:9201")) if err != nil { // Handle error panic(err) } def client.Stop() ``` -------------------------------- ### Terms Aggregation with Date Histogram Sub-aggregation in Go Source: https://context7.com/olivere/elastic/llms.txt Shows how to perform aggregations using the elastic Go client. It creates a terms aggregation grouped by user, with a nested date histogram aggregation by year. The example then processes and prints the results of these aggregations. ```go package main import ( "context" "fmt" "log" "github.com/olivere/elastic/v7" ) func main() { client, err := elastic.NewClient() if err != nil { log.Fatal(err) } ctx := context.Background() // Create aggregations: group by user, with date histogram sub-aggregation timeline := elastic.NewTermsAggregation(). Field("user"). Size(10). OrderByCountDesc() histogram := elastic.NewDateHistogramAggregation(). Field("created"). CalendarInterval("year") timeline = timeline.SubAggregation("history", histogram) // Execute search with aggregation searchResult, err := client.Search(). Index("twitter"). Query(elastic.NewMatchAllQuery()). Aggregation("timeline", timeline). Size(0). Do(ctx) if err != nil { log.Fatalf("Aggregation failed: %v", err) } // Process aggregation results agg, found := searchResult.Aggregations.Terms("timeline") if !found { log.Fatal("Expected aggregation 'timeline' not found") } for _, userBucket := range agg.Buckets { user := userBucket.Key fmt.Printf("User: %v (doc_count: %d)\n", user, userBucket.DocCount) // Process sub-aggregation if hist, found := userBucket.DateHistogram("history"); found { for _, yearBucket := range hist.Buckets { var key string if s := yearBucket.KeyAsString; s != nil { key = *s } fmt.Printf(" Year %s: %d tweets\n", key, yearBucket.DocCount) } } } } ``` -------------------------------- ### Indexing Documents with Go Source: https://context7.com/olivere/elastic/llms.txt Demonstrates how to index documents into Elasticsearch using the oliveder/elastic Go client. It shows indexing using both a Go struct and a raw JSON string. Requires the elastic/v7 package. ```go package main import ( "context" "fmt" "log" "time" "github.com/olivere/elastic/v7" ) type Tweet struct { User string `json:"user"` Message string `json:"message"` Retweets int `json:"retweets"` Created time.Time `json:"created"` Tags []string `json:"tags,omitempty"` } func main() { client, err := elastic.NewClient() if err != nil { log.Fatal(err) } ctx := context.Background() // Index a document using a struct tweet1 := Tweet{ User: "olivere", Message: "Take Five", Retweets: 0, Created: time.Now(), Tags: []string{"music", "jazz"}, } put1, err := client.Index(). Index("twitter"). Id("1"). BodyJson(tweet1). Do(ctx) if err != nil { log.Fatalf("Error indexing document: %v", err) } fmt.Printf("Indexed document %s to index %s (version %d)\n", put1.Id, put1.Index, put1.Version) // Index a document using a JSON string tweet2 := `{"user": "olivere", "message": "It's a Raggy Waltz", "retweets": 5}` put2, err := client.Index(). Index("twitter"). Id("2"). BodyString(tweet2). Do(ctx) if err != nil { log.Fatalf("Error indexing document: %v", err) } fmt.Printf("Indexed document %s (version %d)\n", put2.Id, put2.Version) } ``` -------------------------------- ### Create Elastic Client for Elasticsearch 6.x (Go) Source: https://github.com/olivere/elastic/wiki/Connection-Problems This snippet shows how to create an Elastic client for Elasticsearch version 6.x using the `github.com/olivere/elastic` package. It is compatible with dependency managers like `github.com/golang/dep` and includes error handling for client instantiation. ```go import ( "net/http" "github.com/olivere/elastic" ) // Create a client for Elasticsearch 6.x client, err := elastic.NewClient() if err != nil { // Handle error panic(err) } ```