This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Few sentences can have global significance.

Ethan Zuckerman invented the pop-up commercial in 1997.

He was working for Tripod.com, an online service that let people make little web pages for free. Tripod offered advertising to make money. Advertisers didn't enjoy seeing their advertising next to filthy content, like a user's anal sex website.

Zuckerman's boss wanted a solution. Wasn't there a way to move the ads away from user-generated content?

When you visited a Tripod page, a pop-up ad page appeared. So, the ad isn't officially tied to any user page. It'd float onscreen.

Here’s the thing, though: Zuckerman’s bit of Javascript, that created the popup ad? It was incredibly short — a single line of code:

window.open('http://tripod.com/navbar.html'
"width=200, height=400, toolbar=no, scrollbars=no, resizable=no, target=_top");

Javascript tells the browser to open a 200-by-400-pixel window on top of any other open web pages, without a scrollbar or toolbar.

Simple yet harmful! Soon, commercial websites mimicked Zuckerman's concept, infesting the Internet with pop-up advertising. In the early 2000s, a coder for a download site told me that most of their revenue came from porn pop-up ads.

Pop-up advertising are everywhere. You despise them. Hopefully, your browser blocks them.

Zuckerman wrote a single line of code that made the world worse.

I read Zuckerman's story in How 26 Lines of Code Changed the World. Torie Bosch compiled a humorous anthology of short writings about code that tipped the world.

Most of these samples are quite short. Pop-cultural preconceptions about coding say that important code is vast and expansive. Hollywood depicts programmers as blurs spouting out Niagaras of code. Google's success was formerly attributed to its 2 billion lines of code.

It's usually not true. Google's original breakthrough, the piece of code that propelled Google above its search-engine counterparts, was its PageRank algorithm, which determined a web page's value based on how many other pages connected to it and the quality of those connecting pages. People have written their own Python versions; it's only a few dozen lines.

Google's operations, like any large tech company's, comprise thousands of procedures. So their code base grows. The most impactful code can be brief.

The examples are fascinating and wide-ranging, so read the whole book (or give it to nerds as a present). Charlton McIlwain wrote a chapter on the police beat algorithm developed in the late 1960s to anticipate crime hotspots so law enforcement could dispatch more officers there. It created a racial feedback loop. Since poor Black neighborhoods were already overpoliced compared to white ones, the algorithm directed more policing there, resulting in more arrests, which convinced it to send more police; rinse and repeat.

Kelly Chudler's You Are Not Expected To Understand This depicts the police-beat algorithm.

Even shorter code changed the world: the tracking pixel.

Lily Hay Newman's chapter on monitoring pixels says you probably interact with this code every day. It's a snippet of HTML that embeds a single tiny pixel in an email. Getting an email with a tracking code spies on me. As follows: My browser requests the single-pixel image as soon as I open the mail. My email sender checks to see if Clives browser has requested that pixel. My email sender can tell when I open it.

Adding a tracking pixel to an email is easy:

<img src="URL LINKING TO THE PIXEL ONLINE" width="0" height="0">

An older example: Ellen R. Stofan and Nick Partridge wrote a chapter on Apollo 11's lunar module bailout code. This bailout code operated on the lunar module's tiny on-board computer and was designed to prioritize: If the computer grew overloaded, it would discard all but the most vital work.

When the lunar module approached the moon, the computer became overloaded. The bailout code shut down anything non-essential to landing the module. It shut down certain lunar module display systems, scaring the astronauts. Module landed safely.

22-line code

POODOO    INHINT
    CA  Q
    TS  ALMCADR

    TC  BANKCALL
    CADR  VAC5STOR  # STORE ERASABLES FOR DEBUGGING PURPOSES.

    INDEX  ALMCADR
    CAF  0
ABORT2    TC  BORTENT

OCT77770  OCT  77770    # DONT MOVE
    CA  V37FLBIT  # IS AVERAGE G ON
    MASK  FLAGWRD7
    CCS  A
    TC  WHIMPER -1  # YES.  DONT DO POODOO.  DO BAILOUT.

    TC  DOWNFLAG
    ADRES  STATEFLG

    TC  DOWNFLAG
    ADRES  REINTFLG

    TC  DOWNFLAG
    ADRES  NODOFLAG

    TC  BANKCALL
    CADR  MR.KLEAN
    TC  WHIMPER

This fun book is worth reading.

I'm a contributor to the New York Times Magazine, Wired, and Mother Jones. I've also written Coders: The Making of a New Tribe and the Remaking of the World and Smarter Than You Think: How Technology is Changing Our Minds. Twitter and Instagram: @pomeranian99; Mastodon: @clive@saturation.social.

As technology develops, you should stay up to date

You don't want to fall behind, do you? This post covers 7 tech-related things you should know.

You'll learn how to operate your computer (and other electronic devices) like an expert and how to leverage the Internet and social media to create your brand and business. Read on to stay relevant in today's tech-driven environment.

You must learn how to code.

Future-language is coding. It's how we and computers talk. Learn coding to keep ahead.

Try Codecademy or Code School. There are also numerous free courses like Coursera or Udacity, but they take a long time and aren't necessarily self-paced, so it can be challenging to find the time.

Artificial intelligence (AI) will transform all jobs.

Our skillsets must adapt with technology. AI is a must-know topic. AI will revolutionize every employment due to advances in machine learning.

Here are seven AI subjects you must know.

What is artificial intelligence?

How does artificial intelligence work?

What are some examples of AI applications?

How can I use artificial intelligence in my day-to-day life?

What jobs have a high chance of being replaced by artificial intelligence and how can I prepare for this?

Can machines replace humans? What would happen if they did?

How can we manage the social impact of artificial intelligence and automation on human society and individual people?

Blockchain Is Changing the Future

Few of us know how Bitcoin and blockchain technology function or what impact they will have on our lives. Blockchain offers safe, transparent, tamper-proof transactions.

It may alter everything from business to voting. Seven must-know blockchain topics:

1. Describe blockchain.

2. How does the blockchain function?

3. What advantages does blockchain offer?

4. What possible uses for blockchain are there?

5. What are the dangers of blockchain technology?

6. What are my options for using blockchain technology?

7. What does blockchain technology's future hold?

Cryptocurrencies are here to stay

Cryptocurrencies employ cryptography to safeguard transactions and manage unit creation. Decentralized cryptocurrencies aren't controlled by governments or financial institutions.

Bitcoin, the first cryptocurrency, was launched in 2009. Cryptocurrencies can be bought and sold on decentralized exchanges.

Bitcoin is here to stay.

Bitcoin isn't a fad, despite what some say. Since 2009, Bitcoin's popularity has grown. Bitcoin is worth learning about now. Since 2009, Bitcoin has developed steadily.

With other cryptocurrencies emerging, many people are wondering if Bitcoin still has a bright future. Curiosity is natural. Millions of individuals hope their Bitcoin investments will pay off since they're popular now.

Thankfully, they will. Bitcoin is still running strong a decade after its birth. Here's why.

The Internet of Things (IoT) is no longer just a trendy term.

IoT consists of internet-connected physical items. These items can share data. IoT is young but developing fast.

20 billion IoT-connected devices are expected by 2023. So much data! All IT teams must keep up with quickly expanding technologies. Four must-know IoT topics:

1. Recognize the fundamentals: Priorities first! Before diving into more technical lingo, you should have a fundamental understanding of what an IoT system is. Before exploring how something works, it's crucial to understand what you're working with.

2. Recognize Security: Security does not stand still, even as technology advances at a dizzying pace. As IT professionals, it is our duty to be aware of the ways in which our systems are susceptible to intrusion and to ensure that the necessary precautions are taken to protect them.

3. Be able to discuss cloud computing: The cloud has seen various modifications over the past several years once again. The use of cloud computing is also continually changing. Knowing what kind of cloud computing your firm or clients utilize will enable you to make the appropriate recommendations.

4. Bring Your Own Device (BYOD)/Mobile Device Management (MDM) is a topic worth discussing (MDM). The ability of BYOD and MDM rules to lower expenses while boosting productivity among employees who use these services responsibly is a major factor in their continued growth in popularity.

IoT Security is key

As more gadgets connect, they must be secure. IoT security includes securing devices and encrypting data. Seven IoT security must-knows:

1. fundamental security ideas

2. Authorization and identification

3. Cryptography

4. electronic certificates

5. electronic signatures

6. Private key encryption

7. Public key encryption

Final Thoughts

With so much going on in the globe, it can be hard to stay up with technology. We've produced a list of seven tech must-knows.

Elon's rides start rough, but then...

Elon Musk has never been so hated.

They don’t get Elon.

• He began using PayPal in this manner.

• He began with SpaceX in a similar manner.

• He began with Tesla in this manner.

Disruptive.

Elon had rocky starts. His creativity requires it. Just like writing a first draft.

His fastest way to find the way is to avoid it.

PayPal's pricey launch

PayPal was a 1999 business flop.

They were considered insane.

Elon and his co-founders had big plans for PayPal. They adopted the popular philosophy of the time, exchanging short-term profit for growth, and pulled off a miracle just before the bubble burst.

PayPal was created as a dollar alternative. Original PayPal software allowed PalmPilot money transfers. Unfortunately, there weren't enough PalmPilot users.

Since everyone had email, the company emailed payments. Costs rose faster than sales.

The startup wanted to get a million subscribers by paying $10 to sign up and $10 for each referral. Elon thought the price was fair because PayPal made money by charging transaction fees. They needed to make money quickly.

A Wall Street Journal article valuing PayPal at $500 million attracted investors. The dot-com bubble burst soon after they rushed to get financing.

Musk and his partners sold PayPal to eBay for $1.5 billion in 2002. Musk's most successful company was PayPal.

SpaceX's start-up error

Elon and his friends bought a reconditioned ICBM in Russia in 2002.

He planned to invest much of his wealth in a stunt to promote NASA and space travel.

Many called Elon crazy.

The goal was to buy a cheap Russian rocket to launch mice or plants to Mars and return them. He thought SpaceX would revive global space interest. After a bad meeting in Moscow, Elon decided to build his own rockets to undercut launch contracts.

Then SpaceX was founded.

Elon’s plan was harder than expected.

Explosions followed explosions.

• Millions lost on cargo.

• Millions lost on the rockets.

Investors thought Elon was crazy, but he wasn't.

NASA's biggest competitor became SpaceX. NASA hired SpaceX to handle many of its missions.

Tesla's shaky beginning

Tesla began shakily.

• Clients detested their roadster.

• They continued to miss deadlines.

Lotus would handle the car while Tesla focused on the EV component, easing Tesla's entry. The business experienced elegance creep. Modifying specific parts kept the car from getting worse.

Cost overruns, delays, and other factors changed the Elise-like car's appearance. Only 7% of the Tesla Roadster's parts matched its Lotus twin.

Tesla was about to die.

Elon saved the mess as CEO.

He fired 25% of the workforce to reduce costs.

Elon Musk transformed Tesla into the world's most valuable automaker by running it like a startup.

Tesla hasn't spent a dime on advertising. They let the media do the talking by investing in innovation.

Elon sheds. Elon tries. Elon learns. Elon refines.

Twitter doesn't worry me.

The media is shocked. I’m not.

This is just Elon being Elon.

• Elon makes lean.

• Elon tries new things.

• Elon listens to feedback.

• Elon refines.

Besides Twitter will always be Twitter.

(And you can have it for free)

Master email to sell anything.

Most novice creators don't know how to begin.

Many use online templates. These are usually fluff-filled and niche-specific.

They're robotic and "salesy."

I've attended 3 courses, read 10 books, and sent 600,000 emails in the past five years.

Outcome?

This *proven* email sequence assures me a month's salary every time I send it.

What you will discover in this article is that:

• A full 6-part email sales cycle

• The essential elements you must incorporate

• placeholders and text-filled images

• (Applies to any niche)

This can be a product introduction, holiday, or welcome sequence. This works for email-saleable products.

Let's start

Email 1: Describe your issue

This email is crucial.

How to? We introduce a subscriber or prospect's problem. Later, we'll frame our offer as the solution.

Label the:

• Problem

• Why it still hasn't been fixed

• Resulting implications for the customer

This puts our new subscriber in solve mode and queues our offer:

Email 2: Amplify the consequences

We're still causing problems.

We've created the problem, but now we must employ emotion and storytelling to make it real. We also want to forecast life if nothing changes.

Let's feel:

• What occurs if it is not resolved?

• Why is it crucial to fix it immediately?

• Tell a tale of a person who was in their position. To emphasize the effects, use a true account of another person (or of yourself):

Email 3: Share a transformation story

Selling stories.

Whether in an email, landing page, article, or video. Humanize stories. They give information meaning.

This is where "issue" becomes "solution."

Let's reveal:

• A tale of success

• A new existence and result

• tools and tactics employed

Start by transforming yourself.

Email 4: Prove with testimonials

No one buys what you say.

Emotionally stirred people buy and act. They believe in the product. They feel that if they buy, it will work.

Social proof shows prospects that your solution will help them.

Add:

• Earlier and Later

• Testimonials

• Reviews

Proof this deal works:

Email 5: Reveal your offer

It's showtime.

This is it. Until now, describing the offer and offering links to a landing page have been sparse in the email pictures.

We've been tense. Gaining steam. Building suspense. Email 5 reveals all.

In this email:

• a description of the deal

• A word about a promise

• recapitulation of the transformation

• and make a reference to the urgency Everything should be spelled out clearly:

Email no. 6: Instill urgency

When there are stakes, humans act.

Creating and marketing with haste raises the stakes. Urgency makes a prospect act because they'll miss out or gain immensely.

Urgency converts. Use:

• short time

• Screening

• Scarcity

Urgency and conversions. Limited-time offers are easy.

TL;DR

Use this proven 6-part email sequence (that turns subscribers into profit):

• Introduce a problem

• Amplify it with emotions

• Share transformation story

• Prove it works with testimonials

• Value-stack and present your offer

• Drive urgency and entice the purchase

Looking back into the past can help you understand what's happening today and why.

"Creator economy" conjures up images of originality, sincerity, and passion. Where do Michelangelos and da Vincis push advancement with their gifts without battling for bread and proving themselves posthumously? 

Creativity has been as long as humanity, but it's just recently become a new economic paradigm. We even talk about Web3 now.

Let's examine the creative economy's history to better comprehend it. What brought us here? Looking back can help you understand what's happening now.

No yawning, I promise 😉.

Creator Economy's history

Long, uneven transition to creator economy. Let's examine the economic and societal changes that led us there.

1. Agriculture to industry

Mid-18th-century Industrial Revolution led to shift from agriculture to manufacturing. The industrial economy lasted until World War II.

The industrial economy's principal goal was to provide more affordable, accessible commodities.

Unlike today, products were scarce and inaccessible.

To fulfill its goals, industrialization triggered enormous economic changes, moving power from agrarians to manufacturers. Industrialization brought hard work, rivalry, and new ideas connected to production and automation. Creative thinkers focused on that then.

It doesn't mean music, poetry, or painting had no place back then. They weren't top priority. Artists were independent. The creative field wasn't considered a different economic subdivision.

2. The consumer economy

Manufacturers produced more things than consumers desired after World War II. Stuff was no longer scarce.

The economy must make customers want to buy what the market offers.

The consumer economic paradigm supplanted the industrial one. Customers (or consumers) replaced producers as the new economic center.

Salesmen, marketing, and journalists also played key roles (TV, radio, newspapers, etc.). Mass media greatly boosted demand for goods, defined trends, and changed views regarding nearly everything.

Mass media also gave rise to pop culture, which focuses on mass-market creative products. Design, printing, publishing, multi-media, audio-visual, cinematographic productions, etc. supported pop culture.

The consumer paradigm generated creative occupations and activities, unlike the industrial economy. Creativity was limited by the need for wide appeal.

Most creators were corporate employees.

Creating a following and making a living from it were difficult.

Paul Saffo said that only journalists and TV workers were known. Creators who wished to be known relied on producers, publishers, and other gatekeepers. To win their favor was crucial. Luck was the best tactic.

3. The creative economy

Consumer economy was digitized in the 1990s. IT solutions transformed several economic segments. This new digital economy demanded innovative, digital creativity.

Later, states declared innovation a "valuable asset that creates money and jobs." They also introduced the "creative industries" and the "creative economy" (not creator!) and tasked themselves with supporting them. Australia and the UK were early adopters.

Individual skill, innovation, and intellectual property fueled the creative economy. Its span covered design, writing, audio, video material, etc. The creative economy required IT-powered activity.

The new challenge was to introduce innovations to most economic segments and meet demand for digital products and services.

Despite what the title "creative economy" may imply, it was primarily oriented at meeting consumer needs. It didn't provide inventors any new options to become entrepreneurs. Instead of encouraging innovators to flourish on their own, the creative economy emphasized "employment-based creativity."

4. The creator economy

Next, huge IT platforms like Google, Facebook, YouTube, and others competed with traditional mainstream media.

During the 2008 global financial crisis, these mediums surpassed traditional media. People relied on them for information, knowledge, and networking. That was a digital media revolution. The creator economy started there.

The new economic paradigm aimed to engage and convert clients. The creator economy allowed customers to engage, interact, and provide value, unlike the consumer economy. It gave them instruments to promote themselves as "products" and make money.

Writers, singers, painters, and other creators have a great way to reach fans. Instead of appeasing old-fashioned gatekeepers (producers, casting managers, publishers, etc.), they can use the platforms to express their talent and gain admirers. Barriers fell.

It's not only for pros. Everyone with a laptop and internet can now create.

2022 creator economy:

Since there is no academic description for the current creator economy, we can freestyle.

The current (or Web2) creator economy is fueled by interactive digital platforms, marketplaces, and tools that allow users to access, produce, and monetize content.

No entry hurdles or casting in the creative economy. Sign up and follow platforms' rules. Trick: A platform's algorithm aggregates your data and tracks you. This is the payment for participation.

The platforms offer content creation, design, and ad distribution options. This is platforms' main revenue source.

The creator economy opens many avenues for creators to monetize their work. Artists can now earn money through advertising, tipping, brand sponsorship, affiliate links, streaming, and other digital marketing activities.

Even if your content isn't digital, you can utilize platforms to promote it, interact and convert your audience, and more. No limits. However, some of your income always goes to a platform (well, a huge one).

The creator economy aims to empower online entrepreneurship by offering digital marketing tools and reducing impediments.

Barriers remain. They are just different. Next articles will examine these.

Why update the creator economy for Web3?

I could address this question by listing the present creator economy's difficulties that led us to contemplate a Web3 upgrade.

I don't think these difficulties are the main cause. The mentality shift made us see these challenges and understand there was a better reality without them.

Crypto drove this thinking shift. It promoted disintermediation, independence from third-party service providers, 100% data ownership, and self-sovereignty. Crypto has changed the way we view everyday things.

Crypto's disruptive mission has migrated to other economic segments. It's now called Web3. Web3's creator economy is unique.

Here's the essence of the Web3 economy:

• Eliminating middlemen between creators and fans.

• 100% of creators' data, brand, and effort.

• Business and money-making transparency.

• Authentic originality above ad-driven content.

In the next several articles, I'll explain. We'll also discuss the creator economy and Web3's remedies.

Final thoughts

The creator economy is the organic developmental stage we've reached after all these social and economic transformations.

The Web3 paradigm of the creator economy intends to allow creators to construct their own independent "open economy" and directly monetize it without a third party.

If this approach succeeds, we may enter a new era of wealth creation where producers aren't only the products. New economies will emerge.

This article is a summary. To read the full post, click here.