https://github.com/shaina-gh/token-seperation-using-files
A C++ program for lexical analysis that classifies tokens from source code using file input.
https://github.com/shaina-gh/token-seperation-using-files
compiler-design cpp file token-seperation
Last synced: 11 months ago
JSON representation
A C++ program for lexical analysis that classifies tokens from source code using file input.
- Host: GitHub
- URL: https://github.com/shaina-gh/token-seperation-using-files
- Owner: shaina-gh
- Created: 2025-05-18T16:48:57.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-05-18T16:55:42.000Z (about 1 year ago)
- Last Synced: 2025-05-18T17:48:56.071Z (about 1 year ago)
- Topics: compiler-design, cpp, file, token-seperation
- Language: C++
- Homepage:
- Size: 0 Bytes
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Token Separation Using Files
## 📌 Aim
To develop a C++ program that performs token separation from a given input using file handling. The program identifies and classifies tokens such as **separators**, **operators**, **constants**, and **identifiers** from the input file content.
---
## 🧠 Description
The main objective is to read input from a file, parse each line of text, and classify the tokens into one of the following categories:
- **Separators**: Symbols such as `,`, `;`, `<`, `>`
- **Operators**: Arithmetic and assignment symbols like `=`, `+`, `-`, `*`, `/`
- **Constants**: Numeric values (`0` through `9`)
- **Identifiers**: Any sequence of characters that does not belong to the above categories
The classification logic is implemented using vector-based lookups and simple parsing techniques.
---
## 🛠️ Technologies Used
- Programming Language: **C++**
- Concepts: **File Handling**, **Tokenization**, **Lexical Analysis**
---
## 📄 How It Works
1. The program reads input from a file named `file.txt`.
2. Each line is processed using the `check()` function.
3. Characters in the line are evaluated to determine whether they are:
- **Separators**
- **Operators**
- **Constants**
- **Identifiers**
4. Tokens are printed to the console with their corresponding classification.
---
## 📥 Input
Example content of `file.txt`:
```cpp
int a = b + 5;
```
---
## 📤 Expected Output
```cpp
Identifier int
Identifier a
operator =
Identifier b
operator +
Constant 5
separator ;
```