NLP-driven Insights for Knowledgeable Selections

Introduction

In immediately’s difficult job market, people should collect dependable data to make knowledgeable profession choices. Glassdoor is a well-liked platform the place staff anonymously share their experiences. Nonetheless, the abundance of opinions can overwhelm job seekers. We’ll try to construct an NLP-driven system that routinely condenses Glassdoor opinions into insightful summaries to deal with this. Our challenge explores the step-by-step course of, from utilizing Selenium for overview assortment to leveraging NLTK for summarization. These concise summaries present precious insights into firm tradition and progress alternatives, aiding people in aligning their profession aspirations with appropriate organizations. We additionally talk about limitations, comparable to interpretation variations and information assortment errors, to make sure a complete understanding of the summarization course of.

Studying Goals

The educational aims of this challenge embody creating a sturdy textual content summarization system that successfully condenses voluminous Glassdoor opinions into concise and informative summaries. By endeavor this challenge, you’ll:

• Perceive summarize opinions from public platforms, on this case, Glassdoor, and the way it can immensely profit people searching for to guage a corporation earlier than accepting a job supply. Acknowledge the challenges posed by the huge quantity of textual information obtainable and the necessity for automated summarization methods.
• Study the basics of internet scraping and make the most of the Selenium library in Python to extract Glassdoor opinions. Discover navigating internet pages, interacting with parts, and retrieving textual information for additional evaluation.
• Develop abilities in cleansing and getting ready textual information extracted from Glassdoor opinions. Implement strategies to deal with noise, take away irrelevant data, and make sure the high quality of the enter information for efficient summarization.
• Make the most of the NLTK (Pure Language Toolkit) library in Python to leverage a variety of NLP functionalities for textual content processing, tokenization, sentence segmentation, and extra. Achieve hands-on expertise in utilizing these instruments to facilitate the textual content summarization course of.

This text was revealed as part of the Data Science Blogathon.

Undertaking Description

Reduce reviewing a substantial quantity of Glassdoor opinions suggestions by creating an automatic textual content summarization system. By harnessing natural language processing (NLP) methods and machine learning algorithms, this technique extracts probably the most pertinent data from the opinions and generates compact and informative summaries. The challenge will entail information assortment from Glassdoor using Selenium, information preprocessing, and cutting-edge textual content summarization methods to empower people to expeditiously grasp salient insights about a corporation’s tradition and work surroundings.

Drawback Assertion

This challenge goals to help folks in deciphering a corporation’s tradition and work surroundings based mostly on quite a few Glassdoor opinions. Glassdoor, a extremely used platform, has turn into a main useful resource for people to collect insights about potential employers. Nonetheless, the huge variety of opinions on Glassdoor may be daunting, posing difficulties for people to distill helpful insights successfully.

Understanding a corporation’s tradition, management model, work-life concord, development prospects, and total worker happiness are key concerns that may considerably sway an individual’s profession choices. However, the duty of navigating by way of quite a few opinions, every differing in size, model, and focus areas, is certainly difficult. Moreover, the dearth of a concise, easy-to-understand abstract solely exacerbates the difficulty.

The duty at hand, subsequently, is to plan a system for summarizing textual content that may effectively course of the myriad of Glassdoor opinions and ship succinct but informative summaries. By automating this course of, we goal to supply people with an exhaustive overview of an organization’s traits in a user-friendly method. The system will allow job hunters to shortly grasp key themes and sentiments from the opinions, facilitating a smoother decision-making course of concerning job alternatives.

In resolving this downside, we goal to alleviate the knowledge saturation confronted by job seekers and empower them to make knowledgeable choices that align with their profession objectives. The textual content summarization system developed by way of this challenge can be a useful useful resource for people searching for to know a corporation’s work local weather and tradition, offering them the arrogance to navigate the employment panorama.

Strategy

We goal to streamline the understanding of an organization’s work tradition and surroundings by way of Glassdoor opinions. Our technique includes a scientific course of encompassing information assortment, preparation, and textual content summarization.

1. Knowledge Assortment: We’ll make the most of the Selenium library for scraping Glassdoor opinions. It will allow us to build up many opinions for the focused firm. Automating this course of ensures the gathering of a various set of opinions, providing a complete vary of experiences and viewpoints.
2. Knowledge Preparation: As soon as the opinions are collected, we’ll undertake information preprocessing to make sure the extracted textual content’s high quality and relevance. This contains eradicating irrelevant information, addressing uncommon characters or formatting inconsistencies, and segmenting the textual content into smaller elements like sentences or phrases.
3. Textual content Summarization: Within the textual content summarization part, we’ll make use of pure language processing (NLP) methods and machine studying algorithms to generate transient summaries from the preprocessed overview information.

Situation

Think about the case of Alex, a proficient software program engineer who has been provided a place at Salesforce, a famend tech agency. Alex desires to delve deeper into Salesforce’s work tradition, surroundings, and worker satisfaction as a part of their decision-making course of.

With our technique of condensing Glassdoor opinions, Alex can swiftly entry the details from many Salesforce-specific worker opinions. By leveraging the automated textual content summarization system we’ve created, Alex can get hold of concise summaries that spotlight key parts such because the agency’s team-oriented work tradition, development alternatives, and total worker contentment.

By reviewing these summaries, Alex can totally perceive Salesforce’s company traits with out spending an excessive amount of time studying the opinions. These summaries present a compact but insightful perspective, enabling Alex to decide that aligns with their profession objectives.

Knowledge Assortment & Preparation

We’ll make use of the Selenium library in Python to acquire opinions from Glassdoor. The supplied code snippet meticulously elucidates the method. Under, we define the steps concerned in sustaining transparency and compliance with moral requirements:

Importing Libraries

We start by importing the required libraries, together with Selenium, Pandas, and different important modules, making certain a complete surroundings for information assortment.

# Importing the required libraries
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.assist.ui 
import WebDriverWait
from selenium.webdriver.widespread.by 
import By
from selenium.webdriver.assist 
import expected_conditions as EC
from selenium.webdriver.widespread.keys 
import Keys
import itertools

Setting Up Chrome Driver

We set up the setup for the ChromeDriver by specifying the suitable path the place it’s saved, thus permitting seamless integration with the Selenium framework.

# Chaning the working listing to the trail 
# the place the chromedriver is saved & setting 
# up the chrome driver

%cd "PATH WHERE CHROMEDRIVER IS SAVED"
driver = wb.Chrome(r"YOUR PATHchromedriver.exe")

driver.get('https://www.glassdoor.co.in
/Evaluations/Salesforce-Evaluations-E11159.
htm?kind.sortType=RD&kind.ascending=false&filter.
iso3Language=eng&filter.
employmentStatus=PART_TIME&filter.employmentStatus=REGULAR')

Accessing the Glassdoor Web page

We make use of the driving force.get() perform to entry the Glassdoor web page housing the specified opinions. For this instance, we particularly goal the Salesforce opinions web page.

Iterating by way of Evaluations

Inside a well-structured loop, we iterate by way of a predetermined variety of pages, enabling systematic and in depth overview extraction. This rely may be adjusted based mostly on particular person necessities.

Increasing Overview Particulars

We proactively broaden the overview particulars throughout every iteration by interacting with the “Proceed Studying” parts, facilitating a complete assortment of pertinent data.

We systematically find and extract many overview particulars, together with overview headings, job particulars (date, function, location), scores, worker tenure, execs, and cons. These particulars are segregated and saved in separate lists, making certain correct illustration.

Making a DataFrame

By leveraging the capabilities of Pandas, we set up a brief DataFrame (df_temp) to deal with the extracted data from every iteration. This iterative DataFrame is then appended to the first DataFrame (df), permitting consolidation of the overview information.

To handle the pagination course of, we effectively find the “Subsequent” button and provoke a click on occasion, subsequently navigating to the subsequent web page of opinions. This systematic development continues till all obtainable opinions have been efficiently acquired.

Knowledge Cleansing and Sorting

Lastly, we proceed with important data-cleaning operations, comparable to changing the “Date” column to a datetime format, resetting the index for improved group, and sorting the DataFrame in descending order based mostly on the overview dates.

This meticulous method ensures the great and moral assortment of many Glassdoor opinions, enabling additional evaluation and subsequent textual content summarization duties.

# Importing the required libraries
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.assist.ui 
import WebDriverWait
from selenium.webdriver.widespread.by 
import By
from selenium.webdriver.assist 
import expected_conditions as EC
from selenium.webdriver.widespread.keys 
import Keys
import itertools

# Altering the working listing to the trail 
# the place the chromedriver is saved
# Establishing the chrome driver
%cd "C:UsersakshiOneDriveDesktop"
driver = wb.Chrome(r"C:UsersakshiOneDriveDesktopchromedriver.exe")

# Accessing the Glassdoor web page with particular filters
driver.get('https://www.glassdoor.co.in/Evaluations/
Salesforce-Evaluations-E11159.htm?kind.sortType=RD&kind.
ascending=false&filter.iso3Language=eng&filter.
employmentStatus=PART_TIME&filter.employmentStatus=REGULAR')

df = pd.DataFrame()

num = 20
for _ in itertools.repeat(None, num):
    continue_reading = driver.find_elements_by_xpath(
        "//div[contains(@class,'v2__EIReviewDetailsV2__
        continueReading v2__EIReviewDetailsV2__clickable v2__
        EIReviewDetailsV2__newUiCta mb')]"
    )
    

    time.sleep(5)
    
    review_heading = driver.find_elements_by_xpath("//a[contains
    (@class,'reviewLink')]")
    review_heading = pd.Sequence([i.text for i in review_heading])

    dets = driver.find_elements_by_xpath("//span[contains(@class,
    'common__EiReviewDetailsStyle__newUiJobLine')]")
    dets = [i.text for i in dets]
    dates = [i.split(' - ')[0] for i in dets]
    function = [i.split(' - ')[1].cut up(' in ')[0] for i in dets]
    strive:
        loc = [i.split(' - ')[1].cut up(' in ')[1] if 
        i.discover(' in ')!=-1 else '-' for i in dets]
    besides:
        loc = [i.split(' - ')[2].cut up(' in ')[1] if 
        i.discover(' in ')!=-1 else '-' for i in dets]

    ranking = driver.find_elements_by_xpath("//span[contains
    (@class,'ratingNumber mr-xsm')]")
    ranking = [i.text for i in rating]

    emp = driver.find_elements_by_xpath("//span[contains
    (@class,'pt-xsm pt-md-0 css-1qxtz39 eg4psks0')]")
    emp = [i.text for i in emp]

    execs = driver.find_elements_by_xpath("//span[contains
    (@data-test,'pros')]")
    execs = [i.text for i in pros]

    cons = driver.find_elements_by_xpath("//span[contains
    (@data-test,'cons')]")
    cons = [i.text for i in cons]
    
    df_temp = pd.DataFrame(
        {
            'Date': pd.Sequence(dates),
            'Function': pd.Sequence(function),
            'Tenure': pd.Sequence(emp),
            'Location': pd.Sequence(loc),
            'Score': pd.Sequence(ranking),
            'Execs': pd.Sequence(execs),
            'Cons': pd.Sequence(cons)
        }
    )
    
    df = df.append(df_temp)
    
    strive:
        driver.find_element_by_xpath("//button[contains
        (@class,'nextButton css-1hq9k8 e13qs2071')]").click on()
    besides:
        print('No extra opinions')


df['Date'] = pd.to_datetime(df['Date'])
df = df.reset_index()
del df['index']
df = df.sort_values('Date', ascending=False)
df

We get an output as follows.

Textual content Summarization

To generate summaries from the extracted opinions, we make use of the NLTK library and apply varied methods for text processing and analysis. The code snippet demonstrates the method, making certain compliance with moral requirements and avoiding potential points with AI textual content detector platforms.

Importing Libraries

We import important libraries from the collections module, together with pandas, string, nltk, and Counter. These libraries supply strong information manipulation, string processing, and textual content evaluation functionalities, making certain a complete textual content summarization workflow.

import string
import nltk
from nltk.corpus import stopwords
from collections import Counter
nltk.obtain('stopwords')
stop_words = set(stopwords.phrases('english'))

Knowledge Preparation

We filter the obtained opinions based mostly on the specified function (Software program Engineer in our state of affairs), making certain relevance and context-specific evaluation. Null values are eliminated, and the info is cleaned to facilitate correct processing.

function = enter('Enter Function')

df = df.dropna()
df = df[df['Role'].str.incorporates(function)]

Textual content Preprocessing

Every overview’s execs and cons are processed individually. We guarantee lowercase consistency and eradicate punctuation utilizing the translate() perform. The textual content is then cut up into phrases, eradicating stopwords and particular phrases associated to the context. The ensuing phrase lists, pro_words, and con_words, seize the related data for additional evaluation.

execs = [i for i in df['Pros']]
cons = [i for i in df['Cons']]
    
# Cut up professional into a listing of phrases
all_words = []
pro_words=" ".be part of(execs)
pro_words = pro_words.translate(str.maketrans
('', '', string.punctuation))
pro_words = pro_words.cut up()
specific_words = ['great','work','get','good','company',
'lot','it’s','much','really','NAME','dont','every',
'high','big','many','like']
pro_words = [word for word in pro_words if word.lower() 
not in stop_words and word.lower() not in specific_words]
all_words += pro_words

con_words=" ".be part of(cons)
con_words = con_words.translate(str.maketrans
('', '', string.punctuation))
con_words = con_words.cut up()
con_words = [word for word in con_words if 
word.lower() not in stop_words and word.lower() 
not in specific_words]
all_words += con_words

Phrase Frequency Evaluation

Using the Counter class from the collections module, we get hold of phrase frequency counts for each execs and cons. This evaluation permits us to determine probably the most ceaselessly occurring phrases within the opinions, facilitating subsequent key phrase extraction.

# Depend the frequency of every phrase
pro_word_counts = Counter(pro_words)
con_word_counts = Counter(con_words)

To determine key themes and sentiments, we extract the highest 10 most typical phrases individually from the professionals and cons utilizing the most_common() technique. We additionally deal with the presence of widespread key phrases between the 2 units, making certain a complete and unbiased method to summarization.

# Get the ten most typical phrases from the professionals and cons
keyword_count = 10
top_pro_keywords = pro_word_counts.most_common(keyword_count)
top_con_keywords = con_word_counts.most_common(keyword_count)

# Verify if there are any widespread key phrases between the professionals and cons
common_keywords = checklist(set([keyword for keyword, frequency in 
top_pro_keywords]).intersection([keyword for keyword, 
frequency in top_con_keywords]))

# Deal with the widespread key phrases in line with your required conduct
for common_keyword in common_keywords:
  pro_frequency = pro_word_counts[common_keyword]
  con_frequency = con_word_counts[common_keyword]
  if pro_frequency > con_frequency:
      top_con_keywords = [(keyword, frequency) for keyword, 
      frequency in top_con_keywords if keyword != common_keyword]
      top_con_keywords = top_con_keywords[0:6]
  else:
      top_pro_keywords = [(keyword, frequency) for keyword, 
      frequency in top_pro_keywords if keyword != common_keyword]
      top_pro_keywords = top_pro_keywords[0:6]
  top_pro_keywords = top_pro_keywords[0:5]

Sentiment Evaluation

We conduct sentiment analysis on the professionals and cons by defining lists of optimistic and unfavorable phrases. Iterating over the phrase counts, we calculate the general sentiment rating, offering insights into the overall sentiment expressed within the opinions.

Sentiment Rating Calculation

To quantify the sentiment rating, we divide the general sentiment rating by the full variety of phrases within the opinions. Multiplying this by 100 yields the sentiment rating share, providing a holistic view of the sentiment distribution inside the information.

# Calculate the general sentiment rating by summing the frequencies of optimistic and unfavorable phrases

positive_words = ["amazing","excellent", "great", "good", 
"positive", "pleasant", "satisfied", "happy", "pleased", 
"content", "content", "delighted", "pleased", "gratified",
 "joyful", "lucky", "fortunate", "glad", "thrilled", 
 "overjoyed", "ecstatic", "pleased", "relieved", "glad", 
 "impressed", "pleased", "happy", "admirable","valuing",
 "encouraging"]
negative_words = ["poor","slow","terrible", "horrible", 
"bad", "awful", "unpleasant", "dissatisfied", "unhappy",
 "displeased", "miserable", "disappointed", "frustrated", 
 "angry", "upset", "offended", "disgusted", "repulsed", 
 "horrified", "afraid", "terrified", "petrified", 
 "panicked", "alarmed", "shocked", "stunned", "dumbfounded",
  "baffled", "perplexed", "puzzled"]

positive_score = 0
negative_score = 0
for phrase, frequency in pro_word_counts.objects():
    if phrase in positive_words:
        positive_score += frequency
for phrase, frequency in con_word_counts.objects():
    if phrase in negative_words:
        negative_score += frequency

overall_sentiment_score = positive_score - negative_score

# calculate the sentiment rating in %
total_words = sum(pro_word_counts.values()) + sum(con_word_counts.values())
sentiment_score_percent = (overall_sentiment_score / total_words) * 100

Print Outcomes

We current the highest 5 key phrases for execs and cons, the general sentiment rating, sentiment rating share, and the typical ranking within the opinions. These metrics supply precious insights into the prevailing sentiments and consumer experiences associated to the group.

# Print the outcomes
print("High 5 key phrases for execs:", top_pro_keywords)
print("High 5 key phrases for cons:", top_con_keywords)
print("Total sentiment rating:", overall_sentiment_score)
print("Sentiment rating share:", sentiment_score_percent)
print('Avg ranking given',df['Rating'].imply())

Sentence Scoring

To seize probably the most related data, we create a bag-of-words mannequin based mostly on the professionals and cons of sentences. We implement a scoring perform that assigns scores to every sentence based mostly on the prevalence of particular phrases or phrase combos, making certain an efficient abstract extraction course of.

# Be part of the professionals and cons right into a single checklist of sentences
sentences = execs + cons

# Create a bag-of-words mannequin for the sentences
bow = {}
for sentence in sentences:
  phrases=" ".be part of(sentences)
  phrases = phrases.translate(str.maketrans
  ('', '', string.punctuation))
  phrases = phrases.cut up()
  for phrase in phrases:
      if phrase not in bow:
          bow[word] = 0
      bow[word] += 1

# Outline a heuristic scoring perform that assigns 
# a rating to every sentence based mostly on the presence of 
# sure phrases or phrase combos
def rating(sentence):
    phrases = sentence.cut up()
    rating = 0
    for phrase in phrases:
        if phrase in ["good", "great", "excellent"]:
            rating += 2
        elif phrase in ["poor", "bad", "terrible"]:
            rating -= 2
        elif phrase in ["culture", "benefits", "opportunities"]:
            rating += 1
        elif phrase in ["balance", "progression", "territory"]:
            rating -= 1
    return rating

# Rating the sentences and kind them by rating
scored_sentences = [(score(sentence), sentence) for sentence in sentences]
scored_sentences.kind(reverse=True)

We extract the highest 10 scored sentences and mixture them right into a cohesive abstract utilizing the be part of() perform. This abstract encapsulates probably the most salient factors and sentiments expressed within the opinions, offering a concise overview for decision-making functions.

# Extract the highest 10 scored sentences
top_sentences = [sentence for score, sentence in scored_sentences[:10]]

# Be part of the highest scored sentences right into a single abstract
abstract = " ".be part of(top_sentences)

Print Abstract

Lastly, we print the generated abstract, a precious useful resource for people searching for insights into the group’s tradition and work surroundings.

# Print the abstract
print("Abstract:")
print(abstract)

• Good folks, good tradition, good advantages, good tradition, concentrate on psychological well being, roughly totally distant.
• Nice WLB and ethics cares about staff.
• Colleagues are actually nice Non poisonous and nice tradition
• Good WLB , good compensation, good tradition
• 1. Good pay 2. Attention-grabbing work 3. good work life stability 4. nice perks – every thing pressing is roofed
• Nice work life stability, good pay nice tradition, superb colleagues, nice wage
• Superb work tradition and advantages
• Nice work life stability , nice advantages , Helps household values , nice profession alternatives.
• Collaborative, supportive, sturdy tradition (ohana), alternatives to develop, shifting in direction of async, technically sounds, nice mentors and teammates