Skip to content

README.md

Latest commit

 

History

History
53 lines (38 loc) · 2.19 KB

README.md

File metadata and controls

53 lines (38 loc) · 2.19 KB
Raw

Large File Analyzer (Go)

A high-performance command-line tool built in Go to count word occurrences in large text files. This project focuses on memory-efficient streaming and Go's standard library.

Features

Streaming I/O: Uses bufio.Scanner to process files line-by-line, allowing it to handle files much larger than the available RAM.

  • Case Insensitivity: Matches words regardless of capitalization.

  • Execution Benchmarking: Measures and prints the exact time taken to process the file.

  • CLI Interface: Uses the flag package for a professional terminal experience.

Lessons Learned & "Go Traps"

During the development of this project, I encountered and solved several key Go-specific challenges:

  1. The Immutability of Strings The Mistake: I initially called strings.ToLower(text) and expected the text variable to change. The Lesson: In Go, strings are immutable. Functions like ToLower return a new string. I learned to "capture" the result:
text = strings.ToLower(text)

  1. Efficiency & Loop Optimization The Mistake: I was lowercasing the search word inside the for loop. The Lesson: If a file has 1 million lines, lowercasing the search word 1 million times is a waste of CPU. I moved the transformation outside the loop to optimize performance.

  2. Pointer Dereferencing The Mistake: Using flag.String returns a pointer (*string), not a string. I initially tried to use the pointer directly in string functions. The Lesson: To access the actual value typed by the user, I had to dereference the pointer using the * operator (e.g., *word).

  3. Streaming vs. Loading The Mistake: Thinking os.Open loads the file into memory. The Lesson: os.Open only creates a file descriptor. The bufio.Scanner is the tool that actually "streams" the data piece by piece, preventing the program from crashing on massive files.

How to Use

Build Bash go build -o analyzer Run Place a data.txt file in the directory and run:

./analyzer -word="yourword"

Technical Specs

  • Package os: For file handling and environment pathing.

  • Package bufio: For efficient buffered input.

  • Package strings: For high-level string manipulation.

  • Package time: For precise execution benchmarking.