Large File Analyzer (Go)

A high-performance command-line tool built in Go to count word occurrences in large text files. This project focuses on memory-efficient streaming and Go's standard library.

Features

Streaming I/O: Uses bufio.Scanner to process files line-by-line, allowing it to handle files much larger than the available RAM.

• Case Insensitivity: Matches words regardless of capitalization.

• Execution Benchmarking: Measures and prints the exact time taken to process the file.

• CLI Interface: Uses the flag package for a professional terminal experience.

Lessons Learned & "Go Traps"

During the development of this project, I encountered and solved several key Go-specific challenges:

1. The Immutability of Strings The Mistake: I initially called strings.ToLower(text) and expected the text variable to change. The Lesson: In Go, strings are immutable. Functions like ToLower return a new string. I learned to "capture" the result:

text = strings.ToLower(text)

2. Efficiency & Loop Optimization The Mistake: I was lowercasing the search word inside the for loop. The Lesson: If a file has 1 million lines, lowercasing the search word 1 million times is a waste of CPU. I moved the transformation outside the loop to optimize performance.

3. Pointer Dereferencing The Mistake: Using flag.String returns a pointer (*string), not a string. I initially tried to use the pointer directly in string functions. The Lesson: To access the actual value typed by the user, I had to dereference the pointer using the * operator (e.g., *word).

4. Streaming vs. Loading The Mistake: Thinking os.Open loads the file into memory. The Lesson: os.Open only creates a file descriptor. The bufio.Scanner is the tool that actually "streams" the data piece by piece, preventing the program from crashing on massive files.

How to Use

Build Bash go build -o analyzer Run Place a data.txt file in the directory and run:

./analyzer -word="yourword"

Technical Specs

• Package os: For file handling and environment pathing.

• Package bufio: For efficient buffered input.

• Package strings: For high-level string manipulation.

• Package time: For precise execution benchmarking.