<h1>Advanced Reconnaissance with theHarvester</h1>

Another useful tool for open source intelligence gathering is theHarvester. It is a very simple tool, not as complex as Recon-ng. However, in spite of its simplicity it is very effective in the early stages of a penetration test and it can be used in combination with similar tools (I’ll show you how in a future post).

The tool gathers emails, names, subdomains, IPs, and URLs using multiple public data sources, harvesting a huge quantity of data in an automated way. As we have seen before, this is crucial to determine a company's exposure to the external threat landscape.

theHarvester installation

If you are using Kali Linux, theHarvester comes pre-installed with the official distribution. But at time of writing this post, the version available in the repository (3.1.0) is half broken (it can’t find the APIs). So, it is always a good idea to know how to install it manually.

The generic steps to install theHarvester on Ubuntu 19.04 are as follow:

• Clone the GitHub repository:

git clone https://github.com/laramies/theHarvester.git

• Install the application

pip install -r requirements.txt

You can now start the application either from the CLI or from the menu.

In Kali Linux, theHarvester can be started by navigating in the applications menu by clicking on Applications > Information Gathering > OSINT Analysis > The Harvester

However, you might need to correct the link from the menu.

Install and use alacarte to change the command from “theharvester” to “theHarvester”.

And install the logo, just to look extra cool

Be careful with the installation of several versions. Kali presently ships with two different versions of theHarvester, the default one (3.1.0) accessible via CLI and menu, located at:

But there is also an old version inside /usr/share/golismero/tools/theHarvester

And I installed a third one, included in a recently released framework I’m currently testing, so now my updated script (3.1.1dev) is at:

Anyway, this is just a reminder so that you guys don’t get lost with all these versions and know where to go in the next step.

theHarvester configuration

Like all the other OSINT tools, theHarvester relies heavily in the use of API keys and these are supposed to be in a file called api-keys.yaml

Right now, I have two distinct api-keys files and a symbolic link pointing to the first one. Your setup will probably be different than mine so it’s your job to find the correct file and insert your API keys in it:

And that’s all there is to it.

Using theHarvester

Running theHarvester is pretty straightforward but a few details might make a difference for an advanced user. I’m specifically talking about running the script always from the same location/directory. Why?

Because the directory where you start the command from will be the one where the application will create and save the SQLite database it uses to store the results. Running the app from the menu (unless you change it) will open a CLI in the current user’s root folder.

Obviously, you might want to have separate databases for different entities. If that is the case, then start the script from different directories and you’ll have separate database files.

Creating a separate database for each target

I have a folder under /root/Documents/ for each of my targets. If I run theHarvester from inside the respective folder, a file named stash.sqlite will be created in each of the individual folders. That is theHarvester’s database.

In the end, theHarvester will write all the results inside that database and you can them use them without messing around with other target’s results.

Passive footprinting with theHarvester

So far, in these articles I’ve never directly touched a target. And this tool is perfect for that because it only uses OSINT sources, like search engines, unless you go for the DNS brute force option.

Like we did in Recon-ng, we can focus primarily on either social engineering or on the network infrastructure. In order to do that, all we have to do is select the proper search engines on the command the start the application.

Obviously, the easiest way to use it is just to run all search engines. But it will take longer and sometimes it’s really not useful.

The instructions are pretty clear; there is a set of parameters to be entered as arguments through which we can customize the search. The most important ones are “-d” and “-b” which are mandatory and determine respectively the target domain about which we want to gather information and the data sources we want to use to find them (the list of the sources that can be set is reported in the description).

Take a look at some examples:

Obviously, if you are aiming at finding emails and user accounts, you should focus on the sources with a higher probability of returning that kind of information:

• Baidu
• Yahoo
• IntelX
• Github
• Linkedin
• Twitter

On the other hand, if you are looking only for network information, you should focus on:

• Trello
• OTX

All the other sources will give you a mix of hosts, domains and URLs. Useful, but only after filtered. Anyway, do your own testing and you will see. There are no universal rules, each target will have a completely different attack surface and thus it will require a distinct approach.

Outputting the results

Finally, the tool can immediately provide a report either in html or xml.

theHarvester -d sans.org -b all -f /root/Documents/SANS/Harvest.html

All you have to do is use the proper parameter and the file will be written in the specified folder.

Conclusions

theHarvester is a valuable tool for OSINT because it can quickly discover a good amount of data, especially email addresses. Remember that you need to check ever obtained information. Automatic tools are great but their outputs need to be correctly managed and interpreted.

Remember also of the limits enforced on the free APIs, if you run to many queries you will eventually exhaust your credit and have to wait a few hours or maybe even a day.

Next post: Reconnaissance with Osmedeus