This book is a work in progress – new chapters will be added as they are written. The content of this book targets the latest version of Elasticsearch.
We welcome feedback – if you spot any errors or would like to suggest improvements, please open an issue on the GitHub repo.
The world is swimming in data. For years we have been simply overwhelmed by the quantity of data flowing through and produced by our systems. Existing technology has focused on how to store and structure warehouses full of data. That’s all well and good… until you actually need to make decisions in real-time informed by that data.
Elasticsearch is a distributed scalable real-time search and analytics engine. It enables you to search, analyze and explore your data, often in ways that you did not anticipate at the start of a project. It exists because raw data sitting on a hard drive is just not useful.
Whether you need full text search, real-time analytics of structured data, or a combination of the two, this book will introduce you to the fundamental concepts required to start working with Elasticsearch at a basic level. With these foundations laid, it will move on to more advanced search techniques which you will need to shape the search experience to fit your requirements.
Elasticsearch is not just about full text search. We explain structured search, analytics, the complexities of dealing with human language, geolocation and relationships. We will also discuss how best to model your data to take advantage of the horizontal scalability of Elasticsearch, and how to configure and monitor your cluster when moving to production.
This book is for anybody who wants to put their data to work. It doesn’t matter whether you are starting a new project and have the flexibility to design the system from the ground up, or whether you need to give new life to a legacy system. Elasticsearch will help you to solve existing problems and open the way to new features that you haven’t yet considered.
This book is suitable for novices and experienced users alike. We expect you to have some programming background and, although not required, it would help to have used SQL and a relational database. We explain concepts from first principles, helping novices to gain a sure footing in the complex world of search.
The reader with a search background will also benefit from this book. Elasticsearch is a new technology which has some familiar concepts. The more experienced user will gain an understanding of how those concepts have been implemented and how they interact in the context of Elasticsearch. Even in the early chapters, there are nuggets of information that will be useful to the more advanced user.
Finally, maybe you are in DevOps. While the other departments are stuffing data into Elasticsearch as fast as they can, you’re the one charged with stopping their servers from bursting into flames. Elasticsearch scales effortlessly, as long as your users play within the rules. You need to know how to setup a stable cluster before going into production, then be able to recognize the warning signs at three in the morning in order to prevent catastrophe. The earlier chapters may be of less interest to you but the last part of the book is essential reading — all you need to know to avoid meltdown.
We wrote this book because Elasticsearch needs a narrative. The existing reference documentation is excellent… as long as you know what you are looking for. It assumes that you are intimately familiar with information retrieval concepts, distributed systems, the query DSL and a host of other topics.
This book makes no such assumptions. It has been written so that a complete beginner — to both search and distributed systems — can pick it up and start building a prototype within a few chapters.
We have taken a problem based approach: this is the problem, how do I solve it, and what are the trade-offs of the alternative solutions? We start with the basics and each chapter builds on the preceding ones, providing practical examples and explaining the theory where necessary.
The existing reference documentation explains how to use features. We want this book to explain why and when to use various features.
The explanations and code examples in this book target the latest version of Elasticsearch available at the time of going to print — version 1.4.0 — but Elasticsearch is a rapidly evolving project. The online version of this book will be updated as Elasticsearch changes.
You can find the latest version of this book online at:
You can also track the changes that have been made by visiting the GitHub repository:
Elasticsearch tries very hard to make the complex simple, and to a large degree it succeeds in this. That said, search and distributed systems are complex and sooner or later you have to get to grips with some of the complexity in order to take full advantage of Elasticsearch.
Complexity, however, is not the same as magic. We tend to view complex systems as magical black boxes that respond to incantations, but there are usually simple processes at work within. Understanding these processes helps to disspell the magic — instead of hoping that the black box will do what you want, understanding gives you certainty and clarity.
This is a definitive guide: not only do we help you to get started with Elasticsearch, we tackle the deeper more interesting topics too. These include chapters such as [distributed-cluster], [distributed-docs], [distributed-search], and [inside-a-shard], which are not essential reading but do give you a solid understanding of the internals.
The first part of the book should be read in order as each chapter builds on the previous chapter (although you can skim over the chapters mentioned above). Later chapters like [proximity-matching] and [partial-matching] are more standalone and can be referred to as needed.
This book is divided into six parts:
-
Chapters 1 through 11 provide an introduction to Elasticsearch. They explain how to get your data in and out of Elasticsearch, how Elasticsearch interprets the data in your documents, how basic search works and how to manage indices. By the end of this section you will already be able to integrate your application with Elasticsearch. Chapters 2, 4, 9 and 11 are supplemental chapters which provide more insight into the distributed processes at work, but are not required reading.
-
Chapters 12 through 17 offer a deep dive into search — how to index and query your data to allow you to take advantage of more advanced concepts like word proximity, and partial matching. You will understand how relevance works and how to control it to ensure that the best results are on the first page.
-
Chapters 18 through 24 tackle the thorny subject of dealing with human language through effective use of analyzers and queries. We start with an easy approach to language analysis easily, before diving into the complexities of language, alphabets, and sorting. We cover stemming, stopwords, synonyms, and fuzzy matching.
-
Chapters 25 through 35 discuss aggregations and analytics — ways to summarise and group your data to show overall trends.
-
Chapters 36 through 39 talk about the two approaches to geolocation supported by Elasticsearch: lat/lon geo-points, and complex geo-shapes.
-
Chapters 40 through 43 talk about how to model your data to work most efficiently with Elasticsearch. Representing relationships between entities is not as easy in a search egine as it is in a relational database, which has been designed for that purpose. These chapters also explain how to suit your index design to match the flow of data through your system.
-
Finally, chapters 44 through 45 discuss moving to production: the important configurations, what to monitor, and how to diagnose and prevent problems.
There are three topics which we do not cover in this book, partly because we ran out of space and partly because they are evolving rapidly and anything we write will soon be out of date:
-
Highlighting of result snippets: see {ref}search-request-highlighting.html[Highlighting].
-
Did-you-mean and search-as-you-type suggesters: see {ref}search-suggesters.html[Suggesters].
-
Percolation — finding queries which match a document: see {ref}search-percolate.html[Percolators].
Because this book focuses on problem solving in Elasticsearch rather than syntax, we sometimes reference the existing documentation for a complete list of parameters. The reference documentation can be found here:
The following typographical conventions are used in this book:
- Italic
-
Indicates new terms or concepts.
- Constant width
-
Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords.
Tip
|
This icon signifies a tip, suggestion, or general note. |
Warning
|
This icon indicates a warning or caution. |
Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/oreillymedia/title_title.
This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Elasticsearch - The Definitive Guide by Clinton Gormley and Zachary Tony (O’Reilly). Copyright 2014 Elasticsearch BV, 978-0-596-xxxx-x.”
If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at [email protected].
Note
|
Safari Books Online is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business. |
Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training.
Safari Books Online offers a range of product mixes and pricing programs for organizations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and dozens more. For more information about Safari Books Online, please visit us online.
Please address comments and questions concerning this book to the publisher:
We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://www.oreilly.com/catalog/0636920028505.
To comment or ask technical questions about this book, send email to [email protected].
For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Why are spouses always relegated to a ``last but not least'' disclaimer? There is no doubt in our minds that the two people most deserving of our gratitude are Xavi Sánchez Catalán, Clinton’s long-suffering husband, and Genevieve Flanders, Zach’s fiancée. They have looked after us and loved us, picked up the slack, put up with our absence and our endless moaning about how long the book is taking, and, most importantly, they are still here.
Thank you to Shay Banon for creating Elasticsearch in the first place, and to Elasticsearch the company for supporting our work on the book. Our colleagues at Elasticsearch deserve a big thank you as well. They have helped us to pick through the innards of Elasticsearch to really understand how it works, and they have been responsible for adding improvements and fixing inconsistencies that were brought to light by writing about them.
Two colleagues in particular deserve special mention:
-
Robert Muir patiently shared his deep knowledge of search in general and Lucene in particular. Several chapters are the direct result of joining his pearls of wisdom up into paragraphs.
-
Adrien Grand dived deep into the code to answer question after question, and sense checked our explanations.
Thank you to O’Reilly for undertaking this project and working with us to make this book available online for free, to our editor Brian Anderson for cajoling us along gently, and to our kind and gentle reviewers Benjamin Devèze, Ivan Brusic and Leo Lapworth. Your reassurances kept us hopeful.
Finally, we would like to thank our readers, some of whom we only know by their GitHub identities, who have taken the time to report problems, provide corrections, or suggest improvements:
Adam Canady, Adam Gray, Alexander Reelsen, backer, Baptiste Cabarrou, Bart Vandewoestyne, Brian Wong, Chris Gilmore, Christian Burgas, Colin Goodheart-Smithe, Daniel Wiesmann, David Pilato, Duncan Angus Wilkie, Gavin Foo, geissbock, igal-getrailo, Japh, Jeremy Falling, J.R. Heard, Joe Fleming, Joshua Gourneau, jschneier, Jun Ohtani, klaplume, Kurt Hurtado, lashae, Lucian Precup, Lukáš Vlček, Martijn Dwars, Michael Bonfils, Michael Salmon, Michael Scharf , mitar, Nathan Peck, nobozo, Patrick Peschlow, Paul Schwarz, Pieter Coucke, Robert Muir, Ruslan Zavacky, sangharsh, Santiago Gaviria, submersibletoaster, Ulf, and Will Kahn-Greene.