Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/farooq85-dev/mongodb-aggregation-framework-part-2

This repository is how to use in string operators via aggregation framework in MongoDb.
https://github.com/farooq85-dev/mongodb-aggregation-framework-part-2

database mongodb sql

Last synced: over 1 year ago
JSON representation

This repository is how to use in string operators via aggregation framework in MongoDb.

Awesome Lists containing this project

README 

          
# MongoDb-Aggregation-Framework-Part-2



## This repository is how to use in string operators via aggregation framework in MongoDb.



### What is Aggregation Framework?



#### The Aggregation Framework in MongoDB is a powerful set of tools for performing data transformation and analysis within the database. It allows you to process and manipulate data in complex ways, such as filtering, grouping, sorting, reshaping, and computing aggregates, all within the database itself. This can help offload heavy processing from the application layer and improve performance.



##### Here’s an overview of how the Aggregation Framework works:



1. #### Aggregation Pipeline



   ##### The aggregation framework primarily uses the aggregation pipeline, which consists of a sequence of stages that process data in a series of steps. Each stage takes input, processes it, and passes the result to the next stage. Each stage performs a specific operation, such as filtering, sorting, or transforming documents, and outputs the resulting documents to the next stage. The pipeline follows the "pipe" metaphor, where data flows through a series of transformations.



2. ### Commonly Used Stages in the Aggregation Pipeline



#### Here are some of the most common stages used in an aggregation pipeline:



- $match: Filters documents to pass only those that match the specified condition. This is similar to the find query but can be more advanced.



```

{ $match: { status: "active" } }

```



- $project: Reshapes each document by specifying which fields to include or exclude, and can also add computed fields.



```

  { $project: { name: 1, age: 1, _id: 0 } }

```



- $group: Groups documents by a specified field and performs aggregation operations, like sum, average, count, etc., on other fields.



```

  { $group: { _id: "$category", totalAmount: { $sum: "$amount" } } }

```



- $sort: Sorts the documents in ascending or descending order.



```

{ $sort: { age: 1 } }

```



- $limit: Limits the number of documents passed to the next stage.



```

{ $limit: 10 }

```



- $skip: Skips the specified number of documents.



```

{ $skip: 5 }

```



- $unwind: Deconstructs an array field from the input documents to output a document for each element in the array.



```

{ $unwind: "$items" }

```



- $lookup: Performs a left outer join with another collection, allowing you to combine documents from two collections based on a matching field.



```

{ $lookup: { from: "orders", localField: "_id", foreignField: "customerId", as:

"orderDetails" } }

```



- $addFields (or $set): Adds new fields to documents or modifies existing fields.



```

{ $addFields: { total: { $multiply: ["$price", "$quantity"] } } }

```



3. ### Aggregation Expressions



#### In addition to stages, you can use aggregation expressions to perform complex calculations and transformations on data. These expressions include operators for arithmetic ($sum, $avg, $multiply, etc.), string manipulation ($concat, $substr, $toUpper, etc.), date manipulation ($dateToString, $dateAdd, etc.), conditional logic ($cond, $ifNull), and more.



#### Example of an aggregation expression:



```

{ $project: { totalAmount: { $multiply: ["$price", "$quantity"] } } }

```



4. ### Pipeline Example

   #### Here’s an example of an aggregation pipeline that finds the total sales by product category:



```

db.sales.aggregate([

{ $match: { status: "completed" } },

{ $group: { _id: "$category", totalSales: { $sum: "$amount" } } },

{ $sort: { totalSales: -1 } },

{ $limit: 5 }

])

```



This pipeline:



Filters completed sales ($match).

Groups the sales by category and calculates the total sales for each category ($group).

Sorts the categories by the total sales in descending order ($sort).

Limits the result to the top 5 categories ($limit). 5. $facet (Multistage in Parallel)

The $facet stage allows for running multiple pipelines in parallel within a single aggregation query, enabling you to produce multiple result sets from the same input data.



Example:



```

db.orders.aggregate([

{ $facet: {

    "totalSales": [ { $match: { status: "completed" } }, { $group: { _id: "$category", totalSales: { $sum: "$amount" } } } ],

"latestOrders": [ { $sort: { orderDate: -1 } }, { $limit: 5 } ]

}}

])

```



This produces two result sets: one showing total sales by category and the other showing the latest 5 orders.



6. ### Performance Considerations



   - Indexes: The $match stage can take advantage of indexes for faster querying, so it's a good practice to filter data as early as possible in the pipeline.

   - Memory Limits: Aggregation operations that require large amounts of memory (such as $group) might hit the memory limits. MongoDB uses a memory-mapped sort for these operations, and exceeding the default limit (100MB) can cause the query to fail unless you explicitly enable disk-based sorting with allowDiskUse: true.



7. ### Aggregation in MongoDB Shell vs. Driver



#### The aggregation framework can be used directly in the MongoDB shell or through drivers in various programming languages (e.g., Node.js, Python, Java, etc.). The syntax may vary slightly depending on the driver, but the concepts and stages remain the same.



8. Use Cases



- Reporting and Analytics: Aggregation is commonly used to generate reports, summarize data, and perform statistical analysis.

- Data Transformation: Aggregating data for different formats or aggregating metrics for dashboards.

  ETL Processes: Performing transformations during data extraction or processing in pipelines.



### Summary



#### The Aggregation Framework in MongoDB provides a rich set of tools for transforming, summarizing, and analyzing data directly within the database. It allows you to perform complex operations such as filtering, grouping, sorting, and aggregating data, and it’s especially useful for building data-intensive applications where performance and scalability are critical. By using stages and expressions within the aggregation pipeline, you can execute advanced queries and transformations with high flexibility and efficiency.