Upload readme.txt

Read the readme.txt or something!

readme.txt

@@ -0,0 +1,47 @@
+# Synacknetwork.com's web crawler!
+
+This is a web crawler that scans websites for specified keywords and saves the found URLs in real-time. The crawler simulates both a Googlebot user agent and random user agents for a more human-like browsing experience. 
+(20% of the time is uses google-bot, other 80% it will use random UA)
+
+## Features
+- Crawl websites for specified keywords
+- Use a Googlebot user agent and random user agents to mimic real user behavior
+- Display found URLs and matched keywords in real-time in the UI
+- Save crawled data to a local file and database (local only I'm not sure about HF and Gradio, maybe I'll add a feature later to download found.txt from the UI, if you want this feature msg me or visit synacknetwork and let me know)
+
+## Usage
+1. Enter the starting URL.
+2. Enter keywords (comma-separated) that the crawler should look for.
+3. Choose the crawl depth (the number of levels deep to crawl is 5 max for safety, add more if you want to edit the code on your own space).
+
+## Installation
+Make sure the following Python packages are installed:
+- `requests`
+- `beautifulsoup4`
+- `fake-useragent`
+- `gradio`
+
+These are listed in `requirements.txt`, which will be installed automatically when you run the Space. (DO NOT install sqllite, python handles this on its own, and it will throw errors)
+
+## How It Works
+The crawler simulates both a Googlebot user agent and random user agents for most requests. It checks for the specified keywords in the content of crawled pages and displays the URLs where the keywords were found.
+
+## Example
+- **Start URL**: `https://let's-find-some-cool-shit.common`
+- **Keywords**: `GTA 6 videos, ufo, private videos, my webcam, my files, etc.`
+- **Depth**: `3'
+
+The results will be shown in real-time as the crawler scans through the pages.
+
+## Troubleshooting
+- If you encounter any issues with missing dependencies, make sure your `requirements.txt` is up-to-date.
+- For any errors related to the crawling process, check the logs (accessible in the Space environment).
+
+# Thanks for checking my crawler out!
+I made this mainly to look for Bigfoot, paranormal, and UFO/UAP videos for my YouTube channel. I also used it to find some ring cam videos people have on there personal servers, and they ARE out there play around with keywords have fun!
+contact us @ synacknetwork.com
+have any ideas you would like added 
+please let us know.... 
+
+
+