Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add URL Logging for Visited Pages #9

Open
Cgarg9 opened this issue Nov 1, 2024 · 0 comments
Open

Add URL Logging for Visited Pages #9

Cgarg9 opened this issue Nov 1, 2024 · 0 comments

Comments

@Cgarg9
Copy link
Collaborator

Cgarg9 commented Nov 1, 2024

Description

Currently, the AI Web Scraper (/GenAI/AiWebScraper) does not keep a log of the URLs it visits while retrieving information. To enhance verifiability and allow for a review of the sources, implement a feature that logs all visited URLs in a separate file.

Feature Requirements

  • Save each visited URL in a .txt or .csv file with a timestamp.
  • Ensure that duplicate URLs are not logged if they are visited more than once per session.
  • Log file should be stored within a logs or data subfolder inside the GenAI/AiWebScraper directory.

Acceptance Criteria

  • The feature successfully saves each unique URL visited during a session to the designated log file.
  • The log file is easy to read and structured for future reference.
  • Documentation for this feature is added to the README, explaining the purpose and file location for logged URLs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant