Data scraping from LinkedIn has opportunity to use all job, company and profile data to create all in one source for people who seeks it.
LinkedIn: About Data Scraping
Officially, LinkedIn doesn’t allow the data scraping. LinkedIn wants to protect their user’s data.
But of course it’s legal to scrape public data from any platform. Back in 2019, LinkedIn lost a court case against a startup company over scraping matter.
Basically, who makes any data publicly available there shouldn’t be any expectation of privacy and scraping doesn’t violate any rights.
However LinkedIn still bans direct access of any website based curl attempt with 999 error.
As you can see above many IPs blacklisted by LinkedIn. But there are still IPs can make curl request to LinkedIn.
Solution
To use linkedin.com in our WP Automatic Plugin we prepared a tutorial for job scraping. You need two things:
- Apify account (free) and getting token for be able to use javascript at scraping process.
- LinkedIn account to for getting the login cookie ( li_at cookie ).
Creating Apify Account
We have tutorial here. You have to paste the token to Settings -> Apify API Settings
Also in campaign edit screen you need to activate Apify for site and pages.
Getting the li_at Cookie
For getting the cookie you need an LinkedIn account, you can create here.
After login in the account hit F12 or Right Click -> Inspect
At Application -> Cookies -> linkedin.com find li_at cookie and copy the value.
Finally paste the cookie at campaign options:
Now you can use the visual selector for list item, title and content selection.
Example visual selectors:
- List Item: /html/body/div[6]/div[3]/div[3]/div[2]/div/section[1]/div/div/ul/li[1]/div/div/div[1]/div[2]/div[1]/a
- Single Post: //*[@id=”main-content”]/section[1]/div/section[2]/div/div[1]/div/h1
- Content: //*[@id=”job-details”]
Note: Please do not forget to activate Apify for pages too (as shown above).
