1gn15/censys-scraper
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
2 commits 1 branch 2 tags 43 KiB
Find a file
Exact
1G-N15 c4947fbc93 Add healthcheck endpoint
2025-08-12 17:18:51 +08:00
logs Create Censys Scraper 2025-08-12 16:42:13 +08:00
.env.example Create Censys Scraper 2025-08-12 16:42:13 +08:00
.gitignore Add healthcheck endpoint 2025-08-12 17:18:51 +08:00
.python-version Create Censys Scraper 2025-08-12 16:42:13 +08:00
COPYING Create Censys Scraper 2025-08-12 16:42:13 +08:00
docker-compose.yml Add healthcheck endpoint 2025-08-12 17:18:51 +08:00
Dockerfile Add healthcheck endpoint 2025-08-12 17:18:51 +08:00
main.py Add healthcheck endpoint 2025-08-12 17:18:51 +08:00
README.md Create Censys Scraper 2025-08-12 16:42:13 +08:00

README.md

Censys Scraper

This simple Python server is designed to periodically check if your server's IP address, such as one behind a CloudFlare proxy, has leaked on public databases like Censys. The script uses Censys's search functionality to query for the domain name associated with your server and verify if the IP is present.

Installation

  1. Clone this repository or download the files:
git clone https://codeberg.org/IgnisIncendio/censys-scraper.git
  1. Copy .env.example to .env:
cp .env.example .env
  1. Fill in the required environment variables:
  • CENSYS_SEARCH_URL: The Censys search URL for your server's domain name. Search for your domain name at https://search.censys.io/, and copy paste the generated search URL into this environment variable. An example is provided.
  • IMPORTANT_IP: Your server's IP address, the one that's supposed to be hidden and disassociated from your domain name.
  • SLEEP_DAYS: Number of days between checks. We do not want to overload the servers.
  • GOTIFY_URL: Your Gotify server URL.
  • GOTIFY_TOKEN: Your Gotify app token.
  1. Run the docker container using Docker Compose:
docker compose up -d --build

Notes

Due to the nature of screen-scraping, some requests may fail. In those cases, an error will be logged and we'll wait the specified number of days before retrying.

This script currently logs the HTML and screenshots of (1) the CAPTCHA, (2) the CAPTCHA failure page, if it fails, and (3) the final page if it succeeds. This is mostly for debugging purposes. This may consume quite a lot of disk space.

Contributors

  • 1G-N15

License

Copyright 2025 1G-N15

This file is part of Censys Scraper.

Censys Scraper is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Censys Scraper is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with Censys Scraper. If not, see https://www.gnu.org/licenses/.