Recon with Me !!!

Hi Everyone

In this blog , i will cover automating the enumeration part of reconnaissance and finding bugs using it with the following set of tools.

• Subfinder
• Chaos
• Nuclei
• Httpx
• Notify
• Anew

Subfinder

• Download and install subfinder using the following command

GO111MODULE=on go get -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder

Note: Make sure go1.14+ is installed in your system.

• To run the tool on any of the Target website , use the following command

subfinder -d google.com

• It was able to fetch more than 50246+ subdomains in a short amount of time.

• Then i noticed that we can use -v flag to see the verbose output of the sources that are used to enumerate the subdomains.

subfinder -d google.com -v

• Sources like chaos,threatminer,sublist3r,alienvault ,etc,. are being used to enumerate the subdomains of the target.

• In the Subfinder Github Repository it was mentioned that some of the services will not work until you set it up.

• So i started looking into it to set-up the config-file with the API Keys that are mentioned to see what is the major difference in the results of subdomain

• Navigate to the following directory

cd .config/subfinder/

cat config.yaml to see the config file

• We can see many of the API Key services are Empty , so now are going to fill the necessary API Keys as source for Subdomain Enumeration.

Note The below following API Keys are Free Of Cost and has a Limited number of request in it.

• binaryedge
• censys
• certspotter
• chaos
• dnsdb
• github
• intelx
• passivetotal
• robtex
• securitytrails
• shodan
• spyse
• urlscan
• virustotal
• zoomeye

Binaryedge

1 : Sign up for a free account, verify the account.

2 : Login into the account and Navigate to this URL https://app.binaryedge.io/account/api and give a name to the TOKEN and Click on Generate Token.

Censys

1 : Sign up for a free account, verify the account.

2 : Login into the account and Navigate to this URL https://censys.io/account/api and you will be able to get API ID and Secret

Certspotter

1 : Sign up for a free account.

2 : Login into the account and Navigate to this URL https://sslmate.com/account/api_credentials and you will be able to get the API Key

Note : 100 queries an hour is free.

Chaos

1 : Navigate to this URL https://chaos.projectdiscovery.io/#/

2 : Early access is provided basis on signup and queue and Invite are send out Weekly basis.

3 : Contributor access is Provided on the basis of PR that is done under github.com/projectdiscovery/*.

DNSdb

1 : Sign up for a free community account.

2 : It will ask for Company Email , use Temp Email.

3 : Create an account and verify the email and get the API Key.

Note : It has 30-day renewal (with valid email confirmation)

Github

1 : Sign up for a free account, verify the account.

2 : Navigate to this URL https://github.com/settings/tokens and generate a Personal access tokens.

Intelx

1 : Sign up for a free account, verify the account.

2 : Navigate to this URL https://intelx.io/account?tab=developer and you will get the API details.

Note: Trial 1 week for Free

Passivetotal

1 : Sign up for a free account, verify the account.

2 : Login into the account and Navigate to this URL https://community.riskiq.com/settings and you will be able to get KEY and Secret .

Robtex

1 : Sign in using the google Gmail Account

2 : Navigate to this URL https://www.robtex.com/dashboard/ , you will get the API-Key details.

Security Trails

1 : Sign up for a free account, verify the account.

2 : Login into the account and Navigate to this URL https://securitytrails.com/app/account/credentials and you will be able to get API Key .

Note : Monthly Quoto is 50 API Requests.

Shodan

1 : Register for a shodan account.

2 : Login into the account and navigate to this URL https://account.shodan.io/ , you will get the API Key details.

Spyse

1 : Register for a Spyse account and verify it.

2 : Login into the account and navigate to this URL https://spyse.com/user , you will get the API Token details.

Note : It has 100 API Token valid for 5 days during the Trail Period.

UrlScan

1 : Sign up for a free account, verify the account.

2 : Login into the account and Navigate to this URL https://urlscan.io/user/profile/ and click on Create new API Key.

Virustotal

1 : Register for a Virustotal account and verify it.

2 : Login into the account and navigate to this URL https://www.virustotal.com/gui/user/username/apikey , you will get the API Key details.

Zoom Eye

1 : Register for a ZoomEye account and verify it.

2 : Login into the account and navigate to this URL https://www.zoomeye.org/profile , you will get the API Key details.

Now the Final Config File Looks Full!!!

Now Let us compare the Results Before and After Adding API Keys.

Before API Key

After API Key

• From this we can see we got 114817 Addtional subdomains after using Free API Keys.

• Only difference is the duration that is taken during the API scan is slow than the normal one.

Httpx

• Download and install httpx using the following command

GO111MODULE=on go get -v github.com/projectdiscovery/httpx/cmd/httpx

Note: Make sure go1.14+ is installed in your system.

• To run this tool against all the hosts and subdomains in hosts.txt and returns URLs running HTTP webserver, use the following command

cat hosts.txt | httpx

Nuclei

• Download and install nuclei using the following command

GO111MODULE=on go get -v github.com/projectdiscovery/nuclei/v2/cmd/nuclei

Note: Make sure go1.14+ is installed in your system.

• Scanning for misconfiguration on given list of URLs.

nuclei -l target_urls.txt -t misconfigured-docker.yaml

More Nuclei Templates can be found here https://github.com/projectdiscovery/nuclei-templates.

Notify

• Download and install notify using the following command

GO111MODULE=on go get -v github.com/projectdiscovery/notify/cmd/notify

Note: Make sure go1.14+ is installed in your system.

• Notify also supports piping output of any tool and send it over discord/slack channel as notification.

• Now we need to configure Slack Webhook URL under the notify config file.

• Navigate to the following directory.

cd .config/notify/

cat notify.conf to see the config file.

• From the above image , we can see slack_webhookurl,slack_username,slack_channel,slack are madatory to set of a slack notification using notify

Note : Follow the below steps to create a Slack account and Generate WebhookURL.

Step 1: Sign Up for a slack account and verify the account.

Step 2: During the process , it will ask you to setup the Team Name , Channel Name , Invites.

Step 3: Navigate to this URL https://api.slack.com/apps and click on create app.

Step 4: Enter the App Name and Slack Workspace as shown below.

Step 5: Choose Incoming Webhooks under the App Name and set the Activate Incoming WebhookURL to ON.

Step 6: In the bottom of the page you will be able to see Webhook URLs for Your Workspace, click on Add New Webhook to Workspace.

Step 7: It will ask you for the permission to access slack workspace and the channel name and click on allow.

Step 8: Now your slack webhookurl is generated as shown below.

• We have got the Slack WebhookURL , Slack Username and Channel name.

• Add these into the notify.conf , To edit the file use the following command.

vi .config/notify/notify.conf

• Uncomment the following fields slack_webhook_url, slack_username, slack_channel, slack and add the necessary details.

• Sample command to check the configuration works fine.

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/**********************************

• Scan for subdomains, check alive domains and send slack notifications using notify use the following command.

subfinder -d hackerone.com | httpx | notify

Anew

• Download and install this Tool using the following command

go get -u github.com/tomnomnom/anew

• This tool is used to compare the output with old file and give us what is newly added in the file

For example domain.txt contains a set of sub-domains and now subs.txt same set of subdomains but included few additional extra new subdomains.

• Using the following command we can compare both the files and get what is newly added.

cat subs.txt | anew domain.txt

Security Through Intelligent Automation

• Now by using the above all tools we are going to create a One Linear. subfinder to find new subdomains , check wheather it is alive or not using httpx then scan it with nuclei and send us a slack notification of the output using notify.

First Run

subfinder -silent -dL domains.txt | anew subs.txt

Note : -dL - File containing list of domains to enumerate.

Second Run

while true; do subfinder -dL domains.txt -all | anew subs.txt | httpx | nuclei -t nuclei-templates/ | notify ; sleep 3600; done

Note : while true; do - Keeps the script alive. sleep 3600; done - Run after exactly every one hour.

• The First Run is madatory for this process so that it will do subdomain enumeration for the list of domains that you give and save it in a .txt file.

• The Second Run is used to find new subdomains , check alive status , scan it with nuclei templates and notify us the output.

Note : I highly suggest to run these one linear on VPS system like Digital Ocean or AWS instance.

Why Automation is needed in reconnaissance ?

• The chances of getting duplicates is very less and you will be focusing more on NEW subdomains

For Example

Real Scenario

• I got an alert at 3:54 PM Feb 05, 2021

• I reported the vulnerability to the program at 3:59 PM Feb 05, 2021

• Reported got Triaged and Valid !!!

Huge Shout out to Project Discovery Team for creating Amazing Tools

Kudos to tomnomnom for anew tool and pry0cc for the Onelinear.

Thanks a lot for reading !!!.