This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Consider yourself lucky if your last FAANG interview was rejected.

FAANG—Facebook, Apple, Amazon, Netflix, Google

(I know its manga now, but watch me not care)

These big companies offer many benefits.

1. large salaries and benefits

2. Prestige

3. high expectations for both you and your coworkers.

However, these jobs may have major drawbacks that only become apparent when you're thrown to the wolves, so it's up to you whether you see them as drawbacks or opportunities.

I know most college graduates start working at big tech companies because of their perceived coolness.

I've worked in these companies for years and can tell you what to expect if you get a job here.

Little fish in a vast ocean

The most obvious. Most billion/trillion-dollar companies employ thousands.

You may work on a small, unnoticed product part.

Directors and higher will sometimes make you redo projects they didn't communicate well without respecting your time, talent, or will to work on trivial stuff that doesn't move company needles.

Peers will only say, "Someone has to take out the trash," even though you know company resources are being wasted.

The power imbalance is frustrating.

What you can do about it

Know your WHY. Consider long-term priorities. Though riskier, I stayed in customer-facing teams because I loved building user-facing products.

This increased my impact. However, if you enjoy helping coworkers build products, you may be better suited for an internal team.

I told the Directors and Vice Presidents that their actions could waste Engineering time, even though it was unpopular. Some were receptive, some not.

I kept having tough conversations because they were good for me and the company.

However, some of my coworkers praised my candor but said they'd rather follow the boss.

An outdated piece of technology can take years to update.

Apple introduced Swift for iOS development in 2014. Most large tech companies adopted the new language after five years.

This is frustrating if you want to learn new skills and increase your market value.

Knowing that my lack of Swift practice could hurt me if I changed jobs made writing verbose Objective C painful.

What you can do about it

1. Work on the new technology in side projects; one engineer rewrote the Lyft app in Swift over the course of a weekend and promoted its adoption throughout the entire organization.

2. To integrate new technologies and determine how to combine legacy and modern code, suggest minor changes to the existing codebase.

Most managers spend their entire day in consecutive meetings.

After their last meeting, the last thing they want is another meeting to discuss your career goals.

Sometimes a manager has 15-20 reports, making it hard to communicate your impact.

Misunderstandings and stress can result.

Especially when the manager should focus on selfish parts of the team. Success won't concern them.

What you can do about it

1. Tell your manager that you are a self-starter and that you will pro-actively update them on your progress, especially if they aren't present at the meetings you regularly attend.

2. Keep being proactive and look for mentorship elsewhere if you believe your boss doesn't have enough time to work on your career goals.

3. Alternately, look for a team where the manager has more authority to assist you in making career decisions.

After a certain point, company loyalty can become quite harmful.

Because big tech companies create brand loyalty, too many colleagues stayed in unhealthy environments.

When you work for a well-known company and strangers compliment you, it's fun to tell your friends.

Work defines you. This can make you stay too long even though your career isn't progressing and you're unhappy.

Google may become your surname.

Workplaces are not families.

If you're unhappy, don't stay just because they gave you the paycheck to buy your first home and make you feel like you owe your life to them.

Many employees stayed too long. Though depressed and suicidal.

What you can do about it

1. Your life is not worth a company.

2. Do you want your job title and workplace to be listed on your gravestone? If not, leave if conditions deteriorate.

3. Recognize that change can be challenging. It's difficult to leave a job you've held for a number of years.

4. Ask those who have experienced this change how they handled it.

You still have a bright future if you were rejected from FAANG interviews.

Rejections only lead to amazing opportunities. If you're young and childless, work for a startup.

Companies may pay more than FAANGs. Do your research.

Ask recruiters and hiring managers tough questions about how the company and teams prioritize respectful working hours and boundaries for workers.

I know many 15-year-olds who have a lifelong dream of working at Google, and it saddens me that they're chasing a name on their resume instead of excellence.

This article is not meant to discourage you from working at these companies, but to share my experience about what HR/managers will never mention in interviews.

Read both sides before signing the big offer letter.

No, it’s not smartphones, overeating, or sugar.

Everyday practices affect brain health. Good brain practices increase memory and cognition.

Bad behaviors increase stress, which destroys brain cells.

Bad behaviors can reverse evolution and diminish the brain. So, avoid these practices for brain health.

1. The silent assassin

Introverts appreciated quarantine.

Before the pandemic, they needed excuses to remain home; thereafter, they had enough.

I am an introvert, and I didn’t hate quarantine. There are billions of people like me who avoid people.

Social relationships are important for brain health. Social anxiety harms your brain.

Antisocial behavior changes brains. It lowers IQ and increases drug abuse risk.

What you can do is as follows:

• Make a daily commitment to engage in conversation with a stranger. Who knows, you might turn out to be your lone mate.

• Get outside for at least 30 minutes each day.

• Shop for food locally rather than online.

• Make a call to a friend you haven't spoken to in a while.

2. Try not to rush things.

People love hustle culture. This economy requires a side gig to save money.

Long hours reduce brain health. A side gig is great until you burn out.

Work ages your wallet and intellect. Overworked brains age faster and lose cognitive function.

Working longer hours can help you make extra money, but it can harm your brain.

Side hustle but don't overwork.

What you can do is as follows:

• Decide what hour you are not permitted to work after.

• Three hours prior to night, turn off your laptop.

• Put down your phone and work.

• Assign due dates to each task.

3. Location is everything!

The environment may cause brain fog. High pollution can cause brain damage.

Air pollution raises Alzheimer's risk. Air pollution causes cognitive and behavioral abnormalities.

Polluted air can trigger early development of incurable brain illnesses, not simply lung harm.

Your city's air quality is uncontrollable. You may take steps to improve air quality.

In Delhi, schools and colleges are closed to protect pupils from polluted air. So I've adapted.

What you can do is as follows:

• To keep your mind healthy and young, make an investment in a high-quality air purifier.

• Enclose your windows during the day.

• Use a N95 mask every day.

4. Don't skip this meal.

Fasting intermittently is trendy. Delaying breakfast to finish fasting is frequent.

Some skip breakfast and have a hefty lunch instead.

Skipping breakfast might affect memory and focus. Skipping breakfast causes low cognition, delayed responsiveness, and irritation.

Breakfast affects mood and productivity.

Intermittent fasting doesn't prevent healthy breakfasts.

What you can do is as follows:

• Try to fast for 14 hours, then break it with a nutritious breakfast.

• So that you can have breakfast in the morning, eat dinner early.

• Make sure your breakfast is heavy in fiber and protein.

5. The quickest way to damage the health of your brain

Brain health requires water. 1% dehydration can reduce cognitive ability by 5%.

Cerebral fog and mental clarity might result from 2% brain dehydration. Dehydration shrinks brain cells.

Dehydration causes midday slumps and unproductivity. Water improves work performance.

Dehydration can harm your brain, so drink water throughout the day.

What you can do is as follows:

• Always keep a water bottle at your desk.

• Enjoy some tasty herbal teas.

• With a big glass of water, begin your day.

• Bring your own water bottle when you travel.

Conclusion

Bad habits can harm brain health. Low cognition reduces focus and productivity.

Unproductive work leads to procrastination, failure, and low self-esteem.

Avoid these harmful habits to optimize brain health and function.