This method scrapes websites with ChatGPT (demo with Amazon and Twitter)

In a recent article, I demonstrated how to scrape websites using ChatGPT prompts like scrape website X using Python.

But that doesn’t always work.

After scraping dozens of websites with ChatGPT, I realized that simple prompts rarely work for web scraping.

Using ChatGPT and basic HTML, we can scrape any website.

First things first, use ChatGPT's advanced version (Playground)

ChatGPT Playground's enhanced version is needed to scrape websites quickly. It generates code faster and has less constraints.

Here’s how it looks.

The classic ChatGPT display just allows you to type a prompt. Playground is more customizable and generates code faster than the basic version.

No more delays or restrictions.

In this tutorial, we'll write prompts underneath Playground.

How to Use ChatGPT to Scrape Any Website

To show you how to utilize ChatGPT to scrape any website, we'll use subslikescript, which lists movies.

Later in this post, I'll show you how to scrape Amazon and Twitter, but let's start easy.

If we try a basic prompt like “scrape the movie titles on this website: https://subslikescript.com/movies” it won’t scrape anything. Here’s when we need some HTML knowledge to build the right prompt for ChatGPT.Let's scrape the above movie titles. First, check the webpage. Right-click any element and choose Inspect.

Then see below.

We identify the element to scrape (those in the red box). The data elements are ul, a, and scripts-list.

Here's the prompt for movie titles.

Scrape this website: https://subslikescript.com/movies with Python and Beautiful Soup. Locate the element with tag “ul” and class “scripts-list”. Scrape all the “a” elements inside. Get the text attribute and print it.

Code generated:

import requests
from bs4 import BeautifulSoup

page = requests.get('https://subslikescript.com/movies/')

soup = BeautifulSoup(page.content, 'html.parser')

scripts_list = soup.find(class_="scripts-list")

all_a_elements = scripts_list.find_all('a')

for element in all_a_elements:
    print(element.get_text())

It extracts movie titles successfully.

Let's scrape Amazon and Twitter.

ChatGPT's Amazon scraping

Consider scraping Amazon for self-help books. First, copy the Amazon link for self-help books.

Here’s the link I got. Location-dependent connection. Use my link to replicate my results.

Now we'll check book titles. Here's our element.

If we want to extract the book titles, we need to use the tag name span, class attribute name and a-size-base-plus a-color-base a-text-normalattribute value.

This time I'll use Selenium. I'll add Selenium-specific commands like wait 5 seconds and generate an XPath.

Scrape this website https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10 with Python and Selenium.

Wait 5 seconds and locate all the elements with the following xpath: “span” tag, “class” attribute name, and “a-size-base-plus a-color-base a-text-normal” attribute value. Get the text attribute and print them.

Code generated: (I only had to manually add the path where my chromedriver is located).

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

#initialize webdriver
driver = webdriver.Chrome('<add path of your chromedriver>')

#navigate to the website
driver.get("https://www.amazon.com/s?k=self+help+books&sprefix=self+help+%2Caps%2C158&ref=nb_sb_ss_ts-doa-p_2_10")

#wait 5 seconds to let the page load
sleep(5)

#locate all the elements with the following xpath
elements = driver.find_elements(By.XPATH, '//span[@class="a-size-base-plus a-color-base a-text-normal"]')

#get the text attribute of each element and print it
for element in elements:
    print(element.text)

#close the webdriver
driver.close()

It pulls Amazon book titles.

Utilizing ChatGPT to scrape Twitter

Say you wish to scrape ChatGPT tweets. Search Twitter for ChatGPT and copy the URL.

Here’s the link I got. We must check every tweet. Here's our element.

To extract a tweet, use the div tag and lang attribute.

Again, Selenium.

Scrape this website: https://twitter.com/search?q=chatgpt&src=typed_query using Python, Selenium and chromedriver.

Maximize the window, wait 15 seconds and locate all the elements that have the following XPath: “div” tag, attribute name “lang”. Print the text inside these elements.

Code generated: (again, I had to add the path where my chromedriver is located)

from selenium import webdriver
import time

driver = webdriver.Chrome("/Users/frankandrade/Downloads/chromedriver")
driver.maximize_window()
driver.get("https://twitter.com/search?q=chatgpt&src=typed_query")
time.sleep(15)

elements = driver.find_elements_by_xpath("//div[@lang]")
for element in elements:
    print(element.text)

driver.quit()

You'll get the first 2 or 3 tweets from a search. To scrape additional tweets, click X times.

Congratulations! You scraped websites without coding by using ChatGPT.

Combining AIs creates realistic AI News Videos.

Powerful AI tools like Chat GPT-3 are trending. Have you combined AIs?

The 1-minute fake news video below is startlingly realistic. Artificial Intelligence developed NASA's Mars exploration breakthrough video (AI). However, integrating the aforementioned AIs generated it.

• AI-generated text for the Chat GPT-3 based on a succinct tagline

• DALL-E-2 AI generates an image from a brief slogan.

• Artificial intelligence-generated avatar and speech

This article shows how to use and mix the three AIs to make a realistic news video. First, watch the video (1 minute).

Talk GPT-3

Chat GPT-3 is an OpenAI NLP model. It can auto-complete text and produce conversational responses.

Try it at the playground. The AI will write a comprehensive text from a brief tagline. Let's see what the AI generates with "Breakthrough in Mars Project" as the headline.

Amazing. Our tagline matches our complete and realistic text. Fake news can start here.

DALL-E-2

OpenAI's huge transformer-based language model DALL-E-2. Its GPT-3 basis is geared for image generation. It can generate high-quality photos from a brief phrase and create artwork and images of non-existent objects.

DALL-E-2 can create a news video background. We'll use "Breakthrough in Mars project" again. Our AI creates four striking visuals. Last.

Synthesia

Synthesia lets you quickly produce videos with AI avatars and synthetic vocals.

Avatars are first. Rosie it is.

Upload and select DALL-backdrop. E-2's

Copy the Chat GPT-3 content and choose a synthetic voice.

Voice: English (US) Professional.

Finally, we generate and watch or download our video.

Synthesia AI completes the AI video.

Overview & Resources

We used three AIs to make surprisingly realistic NASA Mars breakthrough fake news in this post. Synthesia generates an avatar and a synthetic voice, therefore it may be four AIs.

These AIs created our fake news.

• AI-generated text for the Chat GPT-3 based on a succinct tagline

• DALL-E-2 AI generates an image from a brief slogan.

• Artificial intelligence-generated avatar and speech

Generative AI hype: some thoughts

The sudden surge in "generative AI" startups and projects feels like the inverse of the recent "web3" boom. Both came from hyped-up pots. But while web3 hyped idealistic tech and an easy way to make money, generative AI hypes unsettling tech and questions whether it can be used to make money.

Web3 is technology looking for problems to solve, while generative AI is technology creating almost too many solutions. Web3 has been evangelists trying to solve old problems with new technology. As Generative AI evolves, users are resolving old problems in stunning new ways.

It's a jab at web3, but it's true. Web3's hype, including crypto, was unhealthy. Always expected a tech crash and shakeout. Tech that won't look like "web3" but will enhance "web2"

But that doesn't mean AI hype is healthy. There'll be plenty of bullshit here, too. As moths to a flame, hype attracts charlatans. Again, the difference is the different starting point. People want to use it. Try it.

With the beta launch of Dall-E 2 earlier this year, a new class of consumer product took off. Midjourney followed suit (despite having to jump through the Discord server hoops). Twelve more generative art projects. Lensa, Prisma Labs' generative AI self-portrait project, may have topped the hype (a startup which has actually been going after this general space for quite a while). This week, ChatGPT went off-topic.

This has a "fake-it-till-you-make-it" vibe. We give these projects too much credit because they create easy illusions. This also unlocks new forms of creativity. And faith in new possibilities.

As a user, it's thrilling. We're just getting started. These projects are not only fun to play with, but each week brings a new breakthrough. As an investor, it's all happening so fast, with so much hype (and ethical and societal questions), that no one knows how it will turn out. Web3's demand won't be the issue. Too much demand may cause servers to melt down, sending costs soaring. Companies will try to mix rapidly evolving tech to meet user demand and create businesses. Frustratingly difficult.

Anyway, I wanted an excuse to post some Lensa selfies.

These are really weird. I recognize them as me or a version of me, but I have no memory of them being taken. It's surreal, out-of-body. Uncanny Valley.

I read 1k+ page 3AC liquidation documentation so you don't have to. Also sharing revised Celsius recovery plans.

3AC's liquidation documents:

Someone disclosed 3AC liquidation records in the BVI courts recently. I'll discuss the leak's timeline and other highlights.

Three Arrows Capital began trading traditional currencies in emerging markets in 2012. They switched to equities and crypto, then purely crypto in 2018.

By 2020, the firm had $703mm in net assets and $1.8bn in loans (these guys really like debt).

The firm's net assets under control reached $3bn in April 2022, according to the filings. 3AC had $600mm of LUNA/UST exposure before May 9th 2022, which put them over.

LUNA and UST go to zero quickly (I wrote about the mechanics of the blowup here). Kyle Davies, 3AC co-founder, told Blockchain.com on May 13 that they have $2.4bn in assets and $2.3bn NAV vs. $2bn in borrowings. As BTC and ETH plunged 33% and 50%, the company became insolvent by mid-2022.

3AC sent $32mm to Tai Ping Shen, a Cayman Islands business owned by Su Zhu and Davies' partner, Kelly Kaili Chen (who knows what is going on here).

3AC had borrowed over $3.5bn in notional principle, with Genesis ($2.4bn) and Voyager ($650mm) having the most exposure.

Genesis demanded $355mm in further collateral in June.

Deribit (another 3AC investment) called for $80 million in mid-June.

Even in mid-June, the corporation was trying to borrow more money to stay afloat. They approached Genesis for another $125mm loan (to pay another lender) and HODLnauts for BTC & ETH loans.

Pretty crazy. 3AC founders used borrowed money to buy a $50 million boat, according to the leak.

Su requesting for $5m + Chen Kaili Kelly asserting they loaned $65m unsecured to 3AC are identified as creditors.

Celsius:

This bankruptcy presentation shows the Celsius breakdown from March to July 14, 2022. From $22bn to $4bn, crypto assets plummeted from $14.6bn to $1.8bn (ouch). $16.5bn in user liabilities dropped to $4.72bn.

In my recent post, I examined if "forced selling" is over, with Celsius' crypto assets being a major overhang. In this presentation, it looks that Chapter 11 will provide clients the opportunity to accept cash at a discount or remain long crypto. Provided that a fresh source of money is unlikely to enter the Celsius situation, cash at a discount or crypto given to customers will likely remain a near-term market risk - cash at a discount will likely come from selling crypto assets, while customers who receive crypto could sell at any time. I'll share any Celsius updates I find.

Conclusion

Only Celsius and the Mt Gox BTC unlock remain as forced selling catalysts. While everything went through a "relief" pump, with ETH up 75% from the bottom and numerous alts multiples higher, there are still macro dangers to equities + risk assets. There's a lot of wealth waiting to be deployed in crypto ($153bn in stables), but fund managers are risk apprehensive (lower than 2008 levels).

We're hopefully over crypto's "bottom," with peak anxiety and forced selling behind us, but we may chop around.

To see the full article, click here.

Many people find the internet's various business models bewildering.

This article summarizes 31 startup e-books.

1. Using the freemium business model (free plus premium),

The freemium business model offers basic software, games, or services for free and charges for enhancements.

Examples include Slack, iCloud, and Google Drive

Provide a rudimentary, free version of your product or service to users.

Google Drive and Dropbox offer 15GB and 2GB of free space but charge for more.

Freemium business model details (Click here)

2. The Business Model of Subscription

Subscription business models sell a product or service for recurring monthly or yearly revenue.

Examples: Tinder, Netflix, Shopify, etc

It's the next step to Freemium if a customer wants to pay monthly for premium features.

Subscription Business Model (Click here)

3. A market-based business strategy

It's an e-commerce site or app where third-party sellers sell products or services.

Examples are Amazon and Fiverr.

• On Amazon's marketplace, a third-party vendor sells a product.

• Freelancers on Fiverr offer specialized skills like graphic design.

Marketplace's business concept is explained.

4. Business plans using aggregates

In the aggregator business model, the service is branded.

Uber, Airbnb, and other examples

Marketplace and Aggregator business models differ.

Amazon and Fiverr link merchants and customers and take a 10-20% revenue split.

Uber and Airbnb-style aggregator Join these businesses and provide their products.

5. The pay-as-you-go concept of business

This is a consumption-based pricing system. Cloud companies use it.

Example: Amazon Web Service and Google Cloud Platform (GCP) (AWS)

AWS, an Amazon subsidiary, offers over 200 pay-as-you-go cloud services.

“In short, the more you use the more you pay”

When it's difficult to divide clients into pricing levels, pay-as-you is employed.

6. The business model known as fee-for-service (FFS)

FFS charges fixed and variable fees for each successful payment.

For instance, PayU, Paypal, and Stripe

Stripe charges 2.9% + 30 per payment.

These firms offer a payment gateway to take consumer payments and deposit them to a business account.

Fintech business model

7. EdTech business strategy

In edtech, you generate money by selling material or teaching as a service.

edtech business models

Freemium When course content is free but certification isn't, e.g. Coursera

FREE TRIAL SkillShare offers free trials followed by monthly or annual subscriptions.

Self-serving marketplace approach where you pick what to learn.

Ad-revenue model The company makes money by showing adverts to its huge user base.

Lock-in business strategy

Lock in prevents customers from switching to a competitor's brand or offering.

It uses switching costs or effort to transmit (soft lock-in), improved brand experience, or incentives.

Apple, SAP, and other examples

Apple offers an iPhone and then locks you in with extra hardware (Watch, Airpod) and platform services (Apple Store, Apple Music, cloud, etc.).

9. Business Model for API Licensing

APIs let third-party apps communicate with your service.

Uber and Airbnb use Google Maps APIs for app navigation.

Examples are Google Map APIs (Map), Sendgrid (Email), and Twilio (SMS).

Business models for APIs

1. Free: The simplest API-driven business model that enables unrestricted API access for app developers. Google Translate and Facebook are two examples.

2. Developer Pays: Under this arrangement, service providers such as AWS, Twilio, Github, Stripe, and others must be paid by application developers.

3. The developer receives payment: These are the compensated content producers or developers who distribute the APIs utilizing their work. For example, Amazon affiliate programs

10. Open-source enterprise

Open-source software can be inspected, modified, and improved by anybody.

For instance, use Firefox, Java, or Android.

Google paid Mozilla $435,702 million to be their primary search engine in 2018.

Open-source software profits in six ways.

1. Paid assistance The Project Manager can charge for customization because he is quite knowledgeable about the codebase.

2. A full database solution is available as a Software as a Service (MongoDB Atlas), but there is a fee for the monitoring tool.

3. Open-core design R studio is a better GUI substitute for open-source applications.

4. sponsors of GitHub Sponsorships benefit the developers in full.

5. demands for paid features Earn Money By Developing Open Source Add-Ons for Current Products

Open-source business model

11. The business model for data

If the software or algorithm collects client data to improve or monetize the system.

Open AI GPT3 gets smarter with use.

Foursquare allows users to exchange check-in locations.

Later, they compiled large datasets to enable retailers like Starbucks launch new outlets.

12. Business Model Using Blockchain

Blockchain is a distributed ledger technology that allows firms to deploy smart contracts without a central authority.

Examples include Alchemy, Solana, and Ethereum.

Business models using blockchain

1. Economy of tokens or utility When a business uses a token business model, it issues some kind of token as one of the ways to compensate token holders or miners. For instance, Solana and Ethereum

2. Bitcoin Cash P2P Business Model Peer-to-peer (P2P) blockchain technology permits direct communication between end users. as in IPFS

3. Enterprise Blockchain as a Service (Baas) BaaS focuses on offering ecosystem services similar to those offered by Amazon (AWS) and Microsoft (Azure) in the web 3 sector. Example: Ethereum Blockchain as a Service with Bitcoin (EBaaS).

4. Blockchain-Based Aggregators With AWS for blockchain, you can use that service by making an API call to your preferred blockchain. As an illustration, Alchemy offers nodes for many blockchains.

13. The free-enterprise model

In the freeterprise business model, free professional accounts are led into the funnel by the free product and later become B2B/enterprise accounts.

For instance, Slack and Zoom

Freeterprise companies flourish through collaboration.

Start with a free professional account to build an enterprise.

14. Business plan for razor blades

It's employed in hardware where one piece is sold at a loss and profits are made through refills or add-ons.

Gillet razor & blades, coffee machine & beans, HP printer & cartridge, etc.

Sony sells the Playstation console at a loss but makes up for it by selling games and charging for online services.

Advantages of the Razor-Razorblade Method

1. lowers the risk a customer will try a product. enables buyers to test the goods and services without having to pay a high initial investment.

2. The product's ongoing revenue stream has the potential to generate sales that much outweigh the original investments.

Razor blade business model

15. The business model of direct-to-consumer (D2C)

In D2C, the company sells directly to the end consumer through its website using a third-party logistic partner.

Examples include GymShark and Kylie Cosmetics.

D2C brands can only expand via websites, marketplaces (Amazon, eBay), etc.

D2C benefits

• Lower reliance on middlemen = greater profitability

• You now have access to more precise demographic and geographic customer data.

• Additional space for product testing

• Increased customisation throughout your entire product line-Inventory Less

16. Business model: White Label vs. Private Label

Private label/White label products are made by a contract or third-party manufacturer.

Most amazon electronics are made in china and white-labeled.

Amazon supplements and electronics.

Contract manufacturers handle everything after brands select product quantities on design labels.

17. The franchise model

The franchisee uses the franchisor's trademark, branding, and business strategy (company).

For instance, KFC, Domino's, etc.

Subway, Domino, Burger King, etc. use this business strategy.

Many people pick a franchise because opening a restaurant is risky.

18. Ad-based business model

Social media and search engine giants exploit search and interest data to deliver adverts.

Google, Meta, TikTok, and Snapchat are some examples.

Users don't pay for the service or product given, e.g. Google users don't pay for searches.

In exchange, they collected data and hyper-personalized adverts to maximize revenue.

19. Business plan for octopuses

Each business unit functions separately but is connected to the main body.

Instance: Oyo

OYO is Asia's Airbnb, operating hotels, co-working, co-living, and vacation houses.

20, Transactional business model, number

Sales to customers produce revenue.

E-commerce sites and online purchases employ SSL.

Goli is an ex-GymShark.

21. The peer-to-peer (P2P) business model

In P2P, two people buy and sell goods and services without a third party or platform.

Consider OLX.

22. P2P lending as a manner of operation

In P2P lending, one private individual (P2P Lender) lends/invests or borrows money from another (P2P Borrower).

Instance: Kabbage

Social lending lets people lend and borrow money directly from each other without an intermediary financial institution.

23. A business model for brokers

Brokerages charge a commission or fee for their services.

Examples include eBay, Coinbase, and Robinhood.

Brokerage businesses are common in Real estate, finance, and online and operate on this model.

1. Buy/sell similar models Examples include financial brokers, insurance brokers, and others who match purchase and sell transactions and charge a commission.

2. These brokers charge an advertiser a fee based on the date, place, size, or type of an advertisement. This is known as the classified-advertiser model. For instance, Craiglist

24. Drop shipping as an industry

Dropshipping allows stores to sell things without holding physical inventories.

When a customer orders, use a third-party supplier and logistic partners.

Retailer product portfolio and customer experience Fulfiller The consumer places the order.

Dropshipping advantages

• Less money is needed (Low overhead-No Inventory or warehousing)

• Simple to start (costs under $100)

• flexible work environment

• New product testing is simpler

25. Business Model for Space as a Service

It's centered on a shared economy that lets millennials live or work in communal areas without ownership or lease.

Consider WeWork and Airbnb.

WeWork helps businesses with real estate, legal compliance, maintenance, and repair.

26. The business model for third-party logistics (3PL)

In 3PL, a business outsources product delivery, warehousing, and fulfillment to an external logistics company.

Examples include Ship Bob, Amazon Fulfillment, and more.

3PL partners warehouse, fulfill, and return inbound and outbound items for a charge.

Inbound logistics involves bringing products from suppliers to your warehouse.

Outbound logistics refers to a company's production line, warehouse, and customer.

27. The last-mile delivery paradigm as a commercial strategy

Last-mile delivery is the collection of supply chain actions that reach the end client.

Examples include Rappi, Gojek, and Postmates.

Last-mile is tied to on-demand and has a nighttime peak.

28. The use of affiliate marketing

Affiliate marketing involves promoting other companies' products and charging commissions.

Examples include Hubspot, Amazon, and Skillshare.

Your favorite youtube channel probably uses these short amazon links to get 5% of sales.

Affiliate marketing's benefits

• In exchange for a success fee or commission, it enables numerous independent marketers to promote on its behalf.

• Ensure system transparency by giving the influencers a specific tracking link and an online dashboard to view their profits.

• Learn about the newest bargains and have access to promotional materials.

29. The business model for virtual goods

This is an in-app purchase for an intangible product.

Examples include PubG, Roblox, Candy Crush, etc.

Consumables are like gaming cash that runs out. Non-consumable products provide a permanent advantage without repeated purchases.

30. Business Models for Cloud Kitchens

Ghost, Dark, Black Box, etc.

Delivery-only restaurant.

These restaurants don't provide dine-in, only delivery.

For instance, NextBite and Faasos

31. Crowdsourcing as a Business Model

Crowdsourcing = Using the crowd as a platform's source.

In crowdsourcing, you get support from people around the world without hiring them.

Crowdsourcing sites

1. Open-Source Software gives access to the software's source code so that developers can edit or enhance it. Examples include Firefox browsers and Linux operating systems.

2. Crowdfunding The oculus headgear would be an example of crowdfunding in essence, with no expectations.