Aggregation Pipeline: From Basic Transformations to Advanced Analytics
Master MongoDB aggregation pipelines for filtering, transformation, grouping, array processing, joins, date analytics, window functions, reporting, and performance tuning using realistic marketplace data.
1. Understanding the Aggregation Framework
Why aggregation is useful
A find() query retrieves documents that match a condition. Real applications often need to calculate totals, group records, combine collections, transform arrays, and create reports. MongoDB's aggregation framework processes documents through an ordered sequence of stages.
A pipeline can filter source documents, reshape them, calculate values, group them, enrich them with related data, and return the final result. A stage's output becomes the next stage's input.
Use find() for straightforward retrieval. Use aggregation when the answer must be computed or transformed across documents. Aggregation is not automatically faster; performance depends on the pipeline, indexes, data volume, and workload.
- Input:
Documents from a collection or another supported source - Processing:
An ordered sequence of aggregation stages - Output:
Transformed documents, calculated results, or written output
A pipeline does not necessarily return one document per input document. $match can reduce the count, $unwind can increase it, and $group can reduce many documents to a few summaries.
db.orders.aggregate([
{ $match: { status: "delivered" } },
{ $sort: { createdAt: -1, _id: -1 } }
]);- Aggregation processes documents through ordered stages.
- Stages can filter, reshape, group, expand, enrich, and write data.
- Aggregation is useful for reporting and analytics.
What is the primary purpose of MongoDB's aggregation framework?
2. Build a Consistent Marketplace Dataset
Dataset used throughout this chapter
The examples use orders, products, and customers. Each order stores a customer reference and an array of purchased items. Each item records its product identifier, quantity, and historical unit price. Historical prices matter because a product's current catalog price may differ from the price originally paid.
The sample data includes several statuses so that filters and business rules can be practiced. Run this in mongosh. If these identifiers already exist, clear the lab collections or choose different identifiers.
Decide whether cancelled orders, shipping fees, discounts, taxes, and refunds are included. Different business rules produce different valid totals.
db.orders.insertMany([
{ _id: 1001, customerId: 501, status: "delivered", createdAt: ISODate("2026-10-01T10:00:00Z"), discount: 100, shippingFee: 50, items: [{ productId: 101, name: "Mechanical Keyboard", category: "Electronics", quantity: 2, price: 2500 }, { productId: 102, name: "USB-C Cable", category: "Accessories", quantity: 3, price: 300 }] },
{ _id: 1002, customerId: 502, status: "shipped", createdAt: ISODate("2026-10-02T12:30:00Z"), discount: 0, shippingFee: 80, items: [{ productId: 103, name: "Monitor", category: "Electronics", quantity: 1, price: 12000 }] },
{ _id: 1003, customerId: 501, status: "delivered", createdAt: ISODate("2026-10-03T09:15:00Z"), discount: 200, shippingFee: 50, items: [{ productId: 104, name: "Office Chair", category: "Furniture", quantity: 1, price: 8000 }, { productId: 102, name: "USB-C Cable", category: "Accessories", quantity: 2, price: 280 }] },
{ _id: 1004, customerId: 503, status: "cancelled", createdAt: ISODate("2026-10-04T14:00:00Z"), discount: 0, shippingFee: 0, items: [{ productId: 101, name: "Mechanical Keyboard", category: "Electronics", quantity: 1, price: 2500 }] },
{ _id: 1005, customerId: 502, status: "delivered", createdAt: ISODate("2026-10-05T08:45:00Z"), discount: 150, shippingFee: 60, items: [{ productId: 105, name: "Desk Lamp", category: "Furniture", quantity: 2, price: 1500 }, { productId: 103, name: "Monitor", category: "Electronics", quantity: 1, price: 11500 }] }
]);
db.products.insertMany([
{ _id: 101, name: "Mechanical Keyboard", category: "Electronics", currentPrice: 2700, stock: 30 },
{ _id: 102, name: "USB-C Cable", category: "Accessories", currentPrice: 350, stock: 200 },
{ _id: 103, name: "Monitor", category: "Electronics", currentPrice: 12500, stock: 15 },
{ _id: 104, name: "Office Chair", category: "Furniture", currentPrice: 8500, stock: 8 },
{ _id: 105, name: "Desk Lamp", category: "Furniture", currentPrice: 1600, stock: 45 }
]);
db.customers.insertMany([
{ _id: 501, name: "Ananya", city: "Chennai", tier: "gold" },
{ _id: 502, name: "Rahul", city: "Bengaluru", tier: "silver" },
{ _id: 503, name: "Priya", city: "Hyderabad", tier: "bronze" }
]);- Order items preserve historical prices.
- Customer and product identifiers connect collections.
- Use BSON Date values for date analytics.
- Define business rules before calculating revenue.
Why preserve the price charged at purchase time in an order?
3. Pipeline Syntax, Stage Order, and Execution
Basic syntax
db.collection.aggregate([stage1, stage2, stage3])
A pipeline is an ordered array of stages. A stage consumes the current pipeline documents and produces documents for the next stage. Moving $match before or after $group can change the meaning: before grouping it filters source records; after grouping it filters summaries.
If $project removes a field, later stages cannot use the removed field unless it is recreated. Use explain() to investigate the execution plan and statistics for important pipelines.
Before adding a stage, identify which fields exist after the preceding stage. This prevents field-path errors.
db.orders.aggregate([
{
$set: {
subtotal: {
$sum: {
$map: {
input: "$items",
as: "item",
in: { $multiply: ["$$item.quantity", "$$item.price"] }
}
}
}
}
},
{ $match: { subtotal: { $gt: 5000 } } }
]);- Pipeline order affects semantics.
- A stage's output is the next stage's input.
- Use explain output to investigate execution.
Can a later stage use a field removed by an earlier $project?
4. Filtering Documents with $match
$match selects documents using familiar MongoDB query predicates, including equality, comparison, logical, array, and existence conditions. Multiple fields in one match object imply AND.
Place selective source filters early when possible. An initial $match may use an appropriate index, but index use depends on query shape, selectivity, available indexes, and the execution plan.
For date ranges, a half-open interval is often convenient: use $gte for the start and $lt for the end. This avoids overlap between adjacent reporting periods.
db.orders.aggregate([
{
$match: {
status: "delivered",
shippingFee: { $gte: 50 },
createdAt: {
$gte: ISODate("2026-10-01T00:00:00Z"),
$lt: ISODate("2026-10-06T00:00:00Z")
}
}
},
{ $project: { customerId: 1, status: 1, createdAt: 1, shippingFee: 1 } }
]);- $match uses query predicates.
- Multiple fields imply AND.
- Early filters can reduce downstream work.
- Use consistent date boundaries.
Where should a selective source filter generally appear if it preserves the required meaning?
5. Reshaping Documents with $project
$project controls which fields appear and can calculate new fields, rename fields, and construct nested output documents. In an inclusion-style projection, _id remains included by default unless explicitly excluded. Do not mix inclusion and exclusion modes except for the special handling of _id.
A dollar-prefixed field path such as "$status" references a field in the current pipeline document. Expressions can be nested to calculate totals. Keep any fields required by later stages in the projection.
An item subtotal is not necessarily net revenue. Discount, shipping, tax, and refund treatment must be defined separately.
db.orders.aggregate([
{
$project: {
_id: 1,
customerId: 1,
status: 1,
itemSubtotal: {
$sum: {
$map: {
input: "$items",
as: "item",
in: { $multiply: ["$$item.quantity", "$$item.price"] }
}
}
},
shippingFee: { $ifNull: ["$shippingFee", 0] }
}
}
]);- $project shapes output fields.
- Field paths use a dollar prefix.
- _id is included by default.
- Keep fields required by later stages.
What does "$price" mean in an aggregation expression?
6. Creating Fields with $set and $addFields
$set and $addFields add or replace fields while preserving other fields in the current document. They are aliases for the same general purpose. Use them to add intermediate calculations without listing every field to retain.
When a field already exists, assigning it in $set replaces its value in the pipeline output. $ifNull can provide a fallback for missing or null fields. A fallback is a business decision: do not automatically convert malformed values to zero when that would hide bad data.
db.orders.aggregate([
{
$set: {
itemSubtotal: {
$sum: {
$map: {
input: "$items",
as: "item",
in: { $multiply: ["$$item.quantity", "$$item.price"] }
}
}
},
discountAmount: { $ifNull: ["$discount", 0] },
shippingAmount: { $ifNull: ["$shippingFee", 0] }
}
},
{
$set: {
totalBeforeTax: {
$add: [{ $subtract: ["$itemSubtotal", "$discountAmount"] }, "$shippingAmount"]
}
}
}
]);- $set preserves other fields.
- Existing values can be replaced.
- $ifNull handles missing and null values.
- Validate types for important calculations.
What does $set do to fields not named in the stage?
7. Expanding Arrays with $unwind
$unwind expands an array so each element produces a separate pipeline document. If an order contains three items, unwinding items produces three documents for that order.
By default, documents with null, missing, or empty arrays do not produce output from $unwind. Use preserveNullAndEmptyArrays: true to retain the parent document in those cases. includeArrayIndex can capture an element's position.
Unwinding can multiply the number of documents processed. Use it when downstream stages need individual array elements; use $map, $filter, or $reduce when the calculation can remain inside the parent document.
For product-level summaries from embedded order items, unwind the array and then group by product identifier.
db.orders.aggregate([
{ $match: { status: "delivered" } },
{ $unwind: "$items" },
{
$project: {
orderId: "$_id",
customerId: 1,
productId: "$items.productId",
productName: "$items.name",
quantity: "$items.quantity",
unitPrice: "$items.price",
lineRevenue: { $multiply: ["$items.quantity", "$items.price"] }
}
}
]);- $unwind emits one document per array element.
- Parent fields are repeated.
- Empty arrays are dropped by default.
- Unwinding can increase processing cost.
How many documents does a normal $unwind produce for an array with four elements?
8. Grouping Documents with $group
$group combines documents with the same _id group key and calculates values with accumulators. Use _id: null to create one overall summary, or a field or constructed document to group by one or multiple dimensions.
Common accumulators include $sum, $avg, $min, $max, $first, $last, $push, $addToSet, $stdDevPop, and $stdDevSamp. $first and $last depend on input order; sort first if chronological order matters.
Grouping can be expensive when many distinct keys are created. Be cautious about collecting very large arrays with $push, especially because result documents must remain within BSON size limits.
Grouping by customer produces customer summaries; grouping by status produces status summaries; grouping by both produces one summary for each combination.
db.orders.aggregate([
{ $match: { status: "delivered" } },
{
$set: {
itemSubtotal: {
$sum: {
$map: {
input: "$items",
as: "item",
in: { $multiply: ["$$item.quantity", "$$item.price"] }
}
}
}
}
},
{
$group: {
_id: "$customerId",
deliveredOrderCount: { $sum: 1 },
totalItemRevenue: { $sum: "$itemSubtotal" },
averageOrderSubtotal: { $avg: "$itemSubtotal" },
lowestOrderSubtotal: { $min: "$itemSubtotal" },
highestOrderSubtotal: { $max: "$itemSubtotal" }
}
},
{ $sort: { totalItemRevenue: -1, _id: 1 } }
]);- $group requires an _id key.
- Use _id: null for one overall summary.
- Accumulators calculate totals and summaries.
- Sort explicitly when $first or $last must be chronological.
Which configuration calculates one total across all input documents?
9. Sorting, Limiting, and Paginating Results
$sort orders documents, $limit restricts the number that continue, and $skip discards the first N documents. Use 1 for ascending and -1 for descending sort order.
For predictable pagination, include a unique tie-breaker such as _id when the main sort field can contain duplicates. Large skip values can be expensive because the server must advance past skipped results. For top-N queries, sort and limit after the calculation that defines the ranking.
A sort before $group can determine the input order for $first and $last; a sort after $group orders the resulting summaries. These have different purposes.
db.orders.aggregate([
{ $match: { status: "delivered" } },
{ $unwind: "$items" },
{
$group: {
_id: "$items.productId",
productName: { $first: "$items.name" },
unitsSold: { $sum: "$items.quantity" },
revenue: { $sum: { $multiply: ["$items.quantity", "$items.price"] } }
}
},
{ $sort: { revenue: -1, _id: 1 } },
{ $limit: 5 }
]);- $sort controls order.
- $limit supports top-N queries.
- Large $skip values can be expensive.
- Use a unique tie-breaker for predictable ordering.
Why add _id as a secondary sort key for pagination?
10. Transforming Arrays with $map, $filter, and $reduce
Use $map to transform every array element, $filter to retain elements that satisfy a condition, and $reduce to accumulate a result across an array. These expressions can preserve one pipeline document per parent document, unlike $unwind.
Use as to name the current element in $map and $filter; reference it with $$variable. In $reduce, $$value is the accumulated result and $$this is the current element.
Array expressions generally expect array input. $ifNull can turn a missing or null array into an empty array, but it does not convert arbitrary non-array values into arrays.
$map transforms elements, $filter selects elements, and $reduce accumulates a result. Use $unwind when later stages need each element as a separate document.
db.orders.aggregate([
{
$set: {
electronicsItems: {
$filter: {
input: { $ifNull: ["$items", []] },
as: "item",
cond: { $eq: ["$$item.category", "Electronics"] }
}
}
}
},
{
$set: {
electronicsSubtotal: {
$reduce: {
input: "$electronicsItems",
initialValue: 0,
in: { $add: ["$$value", { $multiply: ["$$this.quantity", "$$this.price"] }] }
}
}
}
}
]);- $map transforms elements.
- $filter selects elements.
- $reduce accumulates a value.
- Array expressions can avoid document expansion.
Which operator returns only array elements whose category is Electronics?