A homestay search page has to match without diacritics (“nha go da lat” finds “Nhà gỗ Đà Lạt”), filter by guests, amenities, price and a 5 km radius, and sort by distance or relevance, all within a few tens of milliseconds. This post explains how Elasticsearch (ES) stores a document, then redesigns the course project’s homestay mapping and fixes what was wrong; I ran every mapping and query on Elasticsearch 8.15.3 (single node, Docker). Afterwards you’ll be able to pick the right field type for each need and keep ES in sync with PostgreSQL.
Worth reading first: Hands-on EXPLAIN ANALYZE in PostgreSQL (the part on LIKE and pg_trgm), to see where text search in a relational database runs out of steam.
1. When you need Elasticsearch
Key idea: start with PostgreSQL; add ES when you need relevance scoring, facets, language handling, or when search load starts hurting the transactional database.
PostgreSQL covers part of this (ILIKE with pg_trgm, full-text search, PostGIS). As data and query types grow, a search engine becomes the better tool for the job.
Before concluding that PostgreSQL “isn’t enough”, try its own full-text search properly: a generated tsvector column, a GIN index, and the write cost that comes with it (the pending list), measured in Index types beyond the B-tree.
2. Core concepts
Key idea: an index is split into shards, and each shard holds many immutable segments. A document becomes searchable only after the next refresh (every second by default), which is why ES is near real-time.
- Index ≈ table, document (one JSON object) ≈ row, mapping (each field’s type and how it’s indexed) ≈ schema.
- Shard: an index is split into primary shards, each an independent Lucene index. The number of primaries is fixed at index creation.
- Replica: a copy of a primary shard on another node, for fault tolerance and for serving reads.
- Segment: an immutable file inside a shard. New documents go into an in-memory buffer; a refresh writes the buffer out as a new segment and opens it for search. The one-second refresh only applies to indices that received a search in the last 30 seconds.
- Translog: a write log so data not yet flushed to disk survives a node crash.
- Deleted or updated documents are only marked; small segments are merged in the background.
3. Where a document goes
Key idea: each field is stored in the structure that fits its queries: the inverted index for search, doc values for sorting and aggregations, BKD trees for ranges and geo, and _source for returning the original JSON.
The course project’s diagram, redrawn:
flowchart LR
D["Document JSON"] --> T["text fields<br/>name, description"]
D --> K["keyword, numeric, date fields<br/>type, amenities, price"]
D --> G["geo_point field<br/>location"]
D --> S["_source<br/>original JSON"]
T --> A["Analyzer<br/>char filter → tokenizer → token filter"]
A --> II["Inverted index<br/>term → document list"]
K --> N["Normalizer (keyword, optional)"]
N --> II
K --> DV["Doc values<br/>columnar"]
K --> BKD["BKD tree<br/>(numbers, dates)"]
G --> BKD
G --> DV
S --> SF["Stored fields<br/>(.fdt/.fdx)"]
| Structure | Stores | Serves | On by default for |
|---|---|---|---|
Inverted index (.tim, .tip, .doc, .pos) |
term → documents, positions | match, term, match_phrase |
text, keyword |
BKD tree (.kdd, .kdi, .kdm) |
multi-dimensional points | range on numbers and dates, geo_distance, geo_bounding_box |
numeric, date, ip, geo_point |
Doc values (.dvd, .dvm) |
document → value, columnar | sort, aggregations, scripts | every type except text |
Stored fields _source (.fdt, .fdx) |
original JSON, compressed | returning hits, reindexing | every document |
The inverted index is like the index at the back of a book: each word with the pages it appears on. Searching “đà lạt” means looking up two terms and intersecting two lists, with no document scan. But it can’t answer “what’s the price of document 42” (you’d walk every term).
Doc values answer the opposite question: given a document, fetch a field’s value, stored column-wise on disk. Sorting by price, counting by type and averaging price all read doc values. text fields have no doc values by default, so you can’t sort or aggregate on them (section 6).
4. Analyzers: turning text into terms
Key idea: an analyzer is character filters → tokenizer → token filters, applied both at index time and at search time. The standard analyzer keeps Vietnamese diacritics, so diacritic-free search needs asciifolding added yourself.
- Character filters (zero or more): rewrite the character stream, e.g. strip HTML, replace
&withvà(“and”). - Tokenizer (exactly one): split into tokens and record positions.
standardsplits on Unicode word boundaries. - Token filters (zero or more): change tokens, e.g. lowercase, fold diacritics, drop stop words, add synonyms, stem.
The standard analyzer is just the standard tokenizer plus lowercase. Compared with _analyze:
POST homestays_v1/_analyze
{ "analyzer": "standard", "text": "Nhà gỗ Đà Lạt, view đồi thông!" }
→ ["nhà", "gỗ", "đà", "lạt", "view", "đồi", "thông"]
POST homestays_v1/_analyze
{ "analyzer": "vi_folded", "text": "Nhà gỗ Đà Lạt, view đồi thông!" }
→ ["nha", "go", "da", "lat", "view", "doi", "thong"]
vi_folded is a custom analyzer: standard + lowercase + asciifolding (maps accented characters to ASCII, including đ → d). The query “nha go da lat” returns h1 with score 2.647 and h2 (“Căn hộ trung tâm Đà Lạt”, a city-centre apartment) with 0.918.
The search string goes through the same analyzer (or a separate search_analyzer if declared). If the two sides differ, the terms never meet.
A normalizer is the keyword equivalent: no tokenizing, only per-character filters such as lowercase and asciifolding. It lets a term query on city match whether the data says “Đà Lạt” or “da lat”.
If the content itself has translations (a homestay name in Vietnamese and English), give each language its own subfield with its own analyzer; how the database side stores them (a JSON column or a translation table, falling back to a default language) is in Data modeling case studies.
5. Designing the homestay mapping
Key idea: make the mapping strict, pick each type by how the field is used (search, filter, sort, return only), and use one shard for a small index. The course’s original mapping gets three things wrong: float_range for an average price, an array of objects without nested, and 2 shards for a small index.
The original gets a lot right: text with a keyword subfield, a normalizer for property type, geo_point, images with index: false, bulk-loading tips. My rewrite, which I ran:
PUT homestays_v1
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"vi_folded": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] }
},
"normalizer": {
"lower": { "type": "custom", "filter": ["lowercase", "asciifolding"] }
}
}
},
"mappings": {
"dynamic": "strict",
"properties": {
"homestay_id": { "type": "keyword" },
"name": { "type": "text", "analyzer": "vi_folded",
"fields": { "raw": { "type": "keyword", "ignore_above": 256 } } },
"description": { "type": "text", "analyzer": "vi_folded" },
"type": { "type": "keyword", "normalizer": "lower" },
"city": { "type": "keyword", "normalizer": "lower" },
"nightly_price": { "type": "long" },
"price_range": { "type": "long_range" },
"max_guests": { "type": "short" },
"location": { "type": "geo_point" },
"amenities": { "type": "keyword" },
"host": { "properties": {
"host_id": { "type": "keyword" },
"name": { "type": "text" },
"superhost": { "type": "boolean" } } },
"rooms": { "type": "nested", "properties": {
"bed": { "type": "keyword" },
"guests": { "type": "byte" } } },
"images": { "type": "keyword", "index": false, "doc_values": false },
"rating": { "type": "scaled_float", "scaling_factor": 100 },
"created_at": { "type": "date" }
}
}
}
| Field / setting | Choice | Why |
|---|---|---|
dynamic |
strict |
unknown fields are rejected (strict_dynamic_mapping_exception) instead of ES guessing a type |
homestay_id |
keyword |
exact lookups only; term on keyword is usually faster than on numbers (numbers are optimised for range) |
name |
text + name.raw keyword |
search on name, sort and aggregate on name.raw |
nightly_price |
long |
VND has no fractional part |
rating |
scaled_float, scaling_factor: 100 |
4.87 is stored as 487 |
images |
index: false, doc_values: false |
only returned, lives only in _source |
number_of_shards |
1 | a few hundred thousand homestays is only a few GB |
Each choice in detail: dynamic mapping, currency, images, shard size
- Dynamic mapping is handy for experiments but often guesses wrong (numeric strings become
long, dates becometext) and can spawn thousands of fields when data has dynamic keys. - Currencies with fractions: use
scaled_float(stored as an integer after multiplying byscaling_factor, compresses better thanfloatand has no binary rounding surprises in comparisons) or store the smallest unit as along. images: queries on this field are rejected: “Cannot search on field [images] since it is not indexed nor has doc values”.- Shards: Elastic’s docs recommend 10–50 GB per shard and under roughly 200 million documents. Many small shards make every query fan out to more shards and cost heap for metadata.
Range fields are for ranges, not single numbers
Key idea: a range field stores an interval (lower and upper bound) and only fits data that is itself an interval: seasonal price ranges, opening hours, IP ranges. A single number like an average price must be a numeric type.
Experiment: price_range (long_range) with three homestays
Data: h1 700k–1.2M, h2 500k–650k, h3 3M–5M.
Query on price_range |
Result |
|---|---|
range 550k–800k, relation: intersects (price range overlaps the budget) |
h1, h2 |
range 400k–1.3M, relation: within (price range fits entirely in the budget) |
h1, h2 |
sort by price_range |
sort values are meaningless byte strings |
sort by nightly_price |
600000, 850000, 3500000 |
Object vs nested
Key idea: Lucene has no nested objects, so object is flattened into dotted fields. That’s fine for a single object like host; for an array of objects, the link between fields of the same element is lost.
The same room data [{bed: double, guests: 2}, {bed: single, guests: 1}], stored once as an object (rooms_flat) and once as nested (rooms), asking “is there a double room for 1 guest”:
// object: flattened into rooms_flat.bed = [double, single], rooms_flat.guests = [2, 1]
{ "query": { "bool": { "filter": [
{ "term": { "rooms_flat.bed": "double" } },
{ "term": { "rooms_flat.guests": 1 } } ] } } }
→ h1 (wrong: no room is a double for 1 guest)
// nested: each element is its own hidden document
{ "query": { "nested": { "path": "rooms", "query": { "bool": { "filter": [
{ "term": { "rooms.bed": "double" } },
{ "term": { "rooms.guests": 1 } } ] } } } } }
→ (no hits, correct)
Choosing how to store child data: nested, precomputed fields or a separate index
6. Search queries
Key idea: only the text that needs ranking goes in must; every yes/no condition (city, guests, amenities, radius) goes in filter, because filters don’t score and are cached.
The main query of the search page, which I ran:
GET homestays/_search
{
"query": {
"bool": {
"must": [
{ "match": { "name": { "query": "nhà gỗ" } } }
],
"filter": [
{ "term": { "city": "Đà Lạt" } },
{ "range": { "max_guests": { "gte": 2 } } },
{ "terms": { "amenities": ["wifi"] } },
{ "geo_distance": { "distance": "5km", "location": { "lat": 11.9416, "lon": 108.4383 } } }
]
}
},
"sort": [
{ "_geo_distance": { "location": { "lat": 11.9416, "lon": 108.4383 }, "order": "asc", "unit": "m" } }
]
}
Result: h1 with a sort value of 2,180 m.
mustscores with BM25 (a relevance formula based on term frequency);filteronly answers yes/no and is cached per segment.termoncitymatches because thelowernormalizer turns both “Đà Lạt” in the query and “da lat” in the data intoda lat.geo_distanceuses thegeo_pointBKD tree and computes distance on a sphere, so it isn’t distorted like distances measured in Web Mercator (SRID 3857) in the original PostGIS project.- Facets run on doc values: a
termsaggregation ontypegivesapartment 1, cabin 1, villa 1, andavgofnightly_priceis 1,650,000.
Deep pagination: why from + size stops at 10,000
Daily availability doesn’t belong in ES
Key idea: ES filters candidates and returns a few hundred ids; PostgreSQL checks daily availability for exactly those ids and locks the room when booking.
The original project joins homestay_availability (one row per night) to keep only homestays free every night. That calendar changes with every booking, and every change means reindexing the document (with nested, the whole tree). You can put coarser data in ES (fully booked date spans as date_range) to prune, as long as the final check happens in the database.
7. Syncing from PostgreSQL
Key idea: PostgreSQL is the source of truth, ES is a copy. Avoid dual writes; use an outbox or CDC, and make every write to ES safe against duplicates and reordering.
| Approach | How | Problem |
|---|---|---|
| Dual write | the service writes the DB, then calls ES | DB commits but ES fails (or vice versa) and they drift forever; two concurrent requests can write ES out of order |
| Outbox | the same transaction writes the DB and an outbox row; a worker ships it to ES | a little more lag, needs a worker |
| CDC (change data capture) | Debezium reads PostgreSQL’s WAL, publishes events to Kafka, a consumer writes ES | Kafka and Debezium to operate |
- Idempotent writes: use
homestay_idas_id, so writing it again yields the same document. - External versioning: send the row’s version from the DB (
?version=17&version_type=external). ES rejects any version less than or equal to the current one, so a late, stale event can’t overwrite newer data.
I tried it: writing version 17 succeeded, then writing 16 or rewriting 17 both returned version_conflict_engine_exception. The consumer treats that error as “a newer version is already there” and skips it, no retry.
Initial load of millions of documents with _bulk
For the first load (like the course’s data-generation script), Elastic’s docs recommend:
- Use
_bulkwith batches of a few MB to a few tens of MB, growing the size until throughput stops improving. - Set
refresh_interval: -1andnumber_of_replicas: 0during the load, then restore them. - Let ES generate ids if you don’t need fixed ones (it skips the duplicate-id check).
- On
429, back off exponentially.
Exercises
Exercise 1: A mapping for reviews
Task: add reviews. Each review has an author, 1–5 stars, Vietnamese text and a date. You need to search reviews by text, filter by stars, and show the average rating on the homestay search page. Design the mapping.
Solution to Exercise 1: a separate reviews_v1 index
Use a separate reviews_v1 index (alias reviews) rather than nested inside homestays, because reviews are numerous and keep arriving:
PUT reviews_v1
{
"settings": { "analysis": { "analyzer": {
"vi_folded": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] } } } },
"mappings": { "dynamic": "strict", "properties": {
"review_id": { "type": "keyword" },
"homestay_id": { "type": "keyword" },
"author_id": { "type": "keyword" },
"stars": { "type": "byte" },
"content": { "type": "text", "analyzer": "standard",
"fields": { "folded": { "type": "text", "analyzer": "vi_folded" } } },
"created_at": { "type": "date" } } }
}
The average rating and review count are computed in PostgreSQL (or by a periodic aggregation) and written to the homestay document’s rating and review_count, so the search page never queries the review index. content keeps diacritics for accurate ranking, and content.folded drops them for people typing without them. I ran it: multi_match “chu nha than thien” on ["content^2", "content.folded"] found the review “Phòng sạch, chủ nhà thân thiện” (“Clean room, friendly host”).
Exercise 2: Why did this homestay match?
Task: the query “homestay with a room for 4 guests with a double bed” returns a homestay that only has two rooms: {bed: double, guests: 2} and {bed: bunk, guests: 4}. Explain and fix it.
Solution to Exercise 2: the object was flattened
rooms is mapped as object, flattened into rooms.bed = [double, bunk] and rooms.guests = [2, 4], so “bed = double” matches the first element and “guests = 4” matches the second.
Fix: change rooms to nested and use a nested query. Since you can’t change a field type in place, create homestays_v2 with the new mapping, reindex and switch the alias. If this is the only filter you need, precompute a room_types keyword field holding "double:2", "bunk:4" and filter with term on "double:4".
Exercise 3: Review the original mapping
Task: a mapping has avg_price as float_range, host as an object containing a languages array, 2 primary shards for 200,000 documents, and title as text with fields.keyword. Point out the problems.
Solution to Exercise 3: four points in the original mapping
avg_priceis a single number, so it should bescaled_floator alongin the smallest unit; otherwise you can’t sort or aggregate on it.hostis a single object, soobjectis enough.languagesis an array of plain values (keyword), not of objects, so it doesn’t neednestedeither;nestedis only for arrays of objects when you query several fields of the same element.- 200,000 documents is a few GB: 1 primary shard (plus replicas) fits the 10–50 GB per shard guideline.
titlewith akeywordsubfield is right: search ontitle, sort and aggregate ontitle.keyword.
Remember
- Say which queries each structure (inverted index, doc values, BKD tree,
_source) serves. - Explain why
textcan’t be sorted and how a multi-field fixes it. - Know when
nestedis required instead ofobject, and what it costs. - Design a PostgreSQL → ES sync that is safe against duplicate and out-of-order events.
- Know how to change a mapping in production without downtime.
Interview questions
Junior
What is an inverted index?
A structure that maps each term to the list of documents containing it (with positions), like the index at the back of a book.
Searching a word means looking up the term and reading its list instead of scanning every document, which is why full-text search is fast. It’s a poor fit for “what value does this document have” (that’s what doc values are for).
How do text and keyword differ?
text goes through an analyzer for full-text search; keyword keeps the whole string for exact filters, sorting and aggregations.
text is split into tokens and queried with match. keyword can pass through a normalizer and is queried with term. When a field needs both, use a multi-field: name as text and name.raw as keyword.
Mid
What’s inside an analyzer? How do you support Vietnamese search without diacritics?
Character filters (0+) rewrite the string, a tokenizer (exactly 1) splits tokens, token filters (0+) change tokens. For diacritic-free search, use lowercase + asciifolding at both index and search time.
asciifolding also maps đ to d. Keep an extra accented subfield and boost it so correctly accented matches rank first, because folding makes many different words collide.
When do you use nested instead of object?
When the field is an array of objects and you filter several conditions on the same element.
object flattens the array, so conditions match across elements. nested stores each element as its own hidden document, which is correct but costlier: more documents, slower queries, updating one element reindexes the whole parent, and there are size limits.
How do must and filter differ in a bool query?
Both must match, but only must contributes to the relevance score; filter doesn’t score and is cached per segment.
Yes/no conditions (city, price range, radius) belong in filter; only the text that needs ranking goes in must.
Why can’t I find a document right after indexing it?
Because ES is near real-time: a new document is searchable only after the next refresh (every second by default for indices being searched).
Don’t force a refresh after every write, since it creates lots of tiny segments. If you must, use refresh=wait_for on that one request, or read the fresh data from the DB.
Senior
How do you keep Elasticsearch correctly in sync with PostgreSQL?
No dual writes; use an outbox or CDC, write idempotently with _id = primary key plus version_type=external, and reconcile periodically.
- Dual writes: the DB commits but ES fails and they drift forever, and concurrent requests can write out of order.
- An outbox in the same transaction, or CDC (Debezium reading the WAL through Kafka).
- The row version as an external version, so late, stale events are rejected.
- A reconciliation job comparing counts or checksums, and a full reindex from the DB when needed.
You need to change a field’s analyzer on a production index. How?
You can’t change it in place: create a new index, reindex, then swap the alias atomically.
The application must read and write through an alias. Create a new index version with the new mapping and reindex from the old index or reload from the DB; meanwhile the sync pipeline writes to both (or replays changes since the reindex started). Check counts and search quality, swap the alias with _aliases, and keep the old index for a while so you can roll back.
How do you choose the number of shards for an index?
By expected size: 10–50 GB and under roughly 200 million documents per shard; a small index gets 1 primary shard.
Many small shards make every query fan out further and cost heap for metadata. The primary count is fixed at creation and only changes via split, shrink or reindex, so for data that grows over time (logs, events) use time-based indices or data streams with rollover. Scale read load with replicas, not more primaries.
Should daily room availability go into Elasticsearch?
Not as the deciding source: ES filters candidates, PostgreSQL checks daily availability and locks at booking time.
Availability changes with every booking, each change means reindexing the document, and ES lags the DB by seconds, so it can show rooms that are already taken. You can put coarse data in ES to prune, but the final check must happen in the database.
Useful phrases for interviews
- “Full-text queries go to the inverted index; sorting and aggregations read doc values.”
- “We use a custom analyzer with ASCII folding so users can search without diacritics.”
- “Anything that doesn’t affect relevance goes into the filter clause, which is cached and doesn’t score.”
- “Arrays of objects are flattened unless you map them as nested.”
- “The application only talks to an alias, so we can reindex and swap without downtime.”
- “PostgreSQL stays the source of truth; Elasticsearch is a search projection that lags by a few seconds.”
- “We sync through the outbox and use external versioning so late events can’t overwrite newer data.”
- “For deep pagination we use search_after with a point-in-time instead of from and size.”
References
- Elasticsearch: Anatomy of an analyzer
- Elasticsearch: Near real-time search
- Elasticsearch: Nested field type
- Elasticsearch: Range field types
- Elasticsearch: doc_values
- Elasticsearch: Numeric field types
- Elasticsearch: Size your shards
- Elasticsearch: Tune for indexing speed
Notes based on the Advanced Backend course (Ronin Engineer), rewritten in my own words.