How to scrap job postings?

Why scraping job postings is relevant?

Job postings scraping and monitoring is great way to find qualified leads as companies express specific needs inside their job offers on which you can positionned yourself, at the right time.

Several industry use job postings, or hiring intent, to prospect efficiently and contact their prospects when they are ready to buy:

Recruitment agencies

Recruitment agencies are both in need to find candidates and to place them into companies hiring.

Their ideal client is therefore a company in need of a qualified candidates: the most direct way to find these ideal clients is to look at job postings.

Want to place your candidates faster? Read our article about the best sales message for recruitment agencies 👉

IT Services

IT services companies provide IT consulting to companies having software development needs.

A fast way to find new clients is to target companies hiring developers: those are priority leads as they have indeed express an IT development need.

Using job postings is perfect to find leads in need of IT Services along with their technology stack they are explaining in their job description.

Want to find quickly missions for your consultants? Read our article about the best sales message for IT Services companies 👉

Softwares

Inside the job description, companies often describe their technology environment and the tool they are using.

Software companies can target accounts that have a similar technology or competitor software.

Even partners with which they could be intergrated.

Doing so, they can contextualize their prospection automatically using context in found in the job description.

Schools

Schools have to find companies hiring their students.

Their ideal target is a company hiring interns or an apprenticeship position.

Monitoring job postings is the most direct way to find new companies for their students.

Scrap job postings

In this article we are going to see concretly how to scrap job postings monitoring to fuel your sales pipeline with qualified leads!

They are several ways to retrieve automatically job postings, all have their pro and cons, we are going to detail them below.

Scrap Job postings with code

The point of scraping jobs directly from a job boards is to immitate with an algorithm a human query with job title and location, and then parsing the html information rendered by the website.

➕ : The most cost efficient solution.

➖ : The most time consuming solution with high technical skills involves.

To provide you key take aways to start your own job scraper, you will find below python code to retrieve indeed job postings.

import requests
import bs4
from bs4 import BeautifulSoup
import pandas as pd
import time

URL = "https://www.indeed.com/jobs?q=data+scientist+%2420%2C000&l=New+York&start=10"

#conducting a request of the stated URL above:

page = requests.get(URL)

#specifying the desired format of "page" using the html parser - this allows python to read the various components of the page, rather than treating it as one long string.

soup = BeautifulSoup(page.text, "html.parser")

#printing soup in a more structured tree format that makes for easier reading
print(soup.prettify())

def extract_job_title_from_result(soup): 
  jobs = []
  for div in soup.find_all(name="div", attrs={"class":"row"}):
    for a in div.find_all(name="a", attrs={"data-tn-element":"jobTitle"}):
      jobs.append(a["title"])
  return(jobs)
extract_job_title_from_result(soup)

Notice that several steps should be considered if you want to set up automatic job web scraping:

• Host your code in a cloud platform like GCP or AWS with cron starting it regularely,
• Maintain your code as soon as websites are changing,
• Use pool of rotating IPs if you want to get high volume of job postings

Use web scraping platform

Instead of coding your own program to scrap job postings you can use webscraping platforms.

➕ : Come with a price but cost efficient.

➖ : Important time to setup and also technical skills involved to start the solution.

The idea is to indicate to this platform the content you would like to scrap:

1. Select the job board from which you would like to scrap job postings
2. Indicates the scraping process:
   1. Find a list of job postings according to a job title and location,
   2. Indicates to the scraping platform where are located the job postings you would like to scrap
   3. Go through the list of targeted job postings and indicate the different elements you want to retrieve in the job postings: job title, job description, company name, ...
3. Programm the execution of the scraping at frequency you want.

Find below a list of some of the main web scraping tools to scrap job postings:

• Captain data
• Phantom buster
• Instant data scraper (free, but limited features)
• Octoparse

If you want to extract contact info from the companies associated with job postings scraped, you will have to build another scraper - usually these plateforms provide you scraper templates for you to customize.

All in one solution to monitor job postings

The third option is use a solution specialized in job scraping.

Mantiks provides a full automated platform to scrap job postings without technical skills.

First step - Target companies by job postings

To begin with Mantiks, start by creating an account - it's free 🔥

Then create a new report, and specify a job title and location you would like to track.

In the exemple below, we are looking for companies hiring:

• "Sales" or "Business Developer" and not "Intern"
• In the US
• Mentionning "Hubspot" in their job description
• Posted during the last 30 days

Other filters can be used to have a search even more accurate:

• Filter on the number of jobs posted,
• Filter on the company count
• Filter on the company industry
• ...

Second step - find contacts

Going to step 2 on the platform, Mantiks will allow you to find contact in the companies hiring that you targeted.

Select the "Buyer Persona" (your ideal client profile) for which you would like to have the contact, it will include by defaults: civility, full name, job title, email adress, linkedin profile url.

You can add if needed the enrichment of the contact direct phone number.

Third step - Launch and automate your report

Finally, you just have to setup your report name, an email adress to receive your report and the frequency at which you would like to receive it: get the contact of companies hiring every day, every week, and so on.

Once launched, you will receive you report directly by mail with the following information:

• Company details - company name, industry, head count, website, linkedin url.
• Job details - job title, job location, keywords match in the job description, publication date, source url.
• Contact details - civility, full name, job title, email adress, linkedin profile url.

Mantiks helps the sales & marketing teams to automatically identify the contact details of companies that are recruiting.