They depend heavily on open source..

Discord users send billions of messages daily. Users wish to search these messages. How do we index these to search by message keywords?

Let’s find out.

1. Discord utilizes Elasticsearch. Elasticsearch is a free, open search engine for textual, numerical, geographical, structured, and unstructured data. Apache Lucene powers Elasticsearch.
   

2. How does elastic search store data? It stores it as numerous key-value pairs in JSON documents.
   

3. How does elastic search index? Elastic search's index is inverted. An inverted index lists every unique word in every page and where it appears.

4. Elasticsearch indexes documents and generates an inverted index to make data searchable in near real-time. The index API adds or updates JSON documents in a given index.

1. Let's examine how discord uses Elastic Search. Elasticsearch prefers bulk indexing. Discord couldn't index real-time messages. You can't search posted messages. You want outdated messages.

6. Let's check what bulk indexing requires.
1. A temporary queue for incoming communications.
2. Indexer workers that index messages into elastic search.

1. Discord's queue is Celery. The queue is open-source. Elastic search won't run on a single server. It's clustered. Where should a message go? Where?

8. A shard allocator decides where to put the message. Nevertheless. Shattered? A shard combines elastic search and index on. So, these two form a shard which is used as a unit by discord. The elastic search itself has some shards. But this is different, so don’t get confused.

1. Now, the final part is service discovery — to discover the elastic search clusters and the hosts within that cluster. This, they do with the help of etcd another open source tool.
   

A great thing to notice here is that discord relies heavily on open source systems and their base implementations which is very different from a lot of other products.

Generative AI hype: some thoughts

The sudden surge in "generative AI" startups and projects feels like the inverse of the recent "web3" boom. Both came from hyped-up pots. But while web3 hyped idealistic tech and an easy way to make money, generative AI hypes unsettling tech and questions whether it can be used to make money.

Web3 is technology looking for problems to solve, while generative AI is technology creating almost too many solutions. Web3 has been evangelists trying to solve old problems with new technology. As Generative AI evolves, users are resolving old problems in stunning new ways.

It's a jab at web3, but it's true. Web3's hype, including crypto, was unhealthy. Always expected a tech crash and shakeout. Tech that won't look like "web3" but will enhance "web2"

But that doesn't mean AI hype is healthy. There'll be plenty of bullshit here, too. As moths to a flame, hype attracts charlatans. Again, the difference is the different starting point. People want to use it. Try it.

With the beta launch of Dall-E 2 earlier this year, a new class of consumer product took off. Midjourney followed suit (despite having to jump through the Discord server hoops). Twelve more generative art projects. Lensa, Prisma Labs' generative AI self-portrait project, may have topped the hype (a startup which has actually been going after this general space for quite a while). This week, ChatGPT went off-topic.

This has a "fake-it-till-you-make-it" vibe. We give these projects too much credit because they create easy illusions. This also unlocks new forms of creativity. And faith in new possibilities.

As a user, it's thrilling. We're just getting started. These projects are not only fun to play with, but each week brings a new breakthrough. As an investor, it's all happening so fast, with so much hype (and ethical and societal questions), that no one knows how it will turn out. Web3's demand won't be the issue. Too much demand may cause servers to melt down, sending costs soaring. Companies will try to mix rapidly evolving tech to meet user demand and create businesses. Frustratingly difficult.

Anyway, I wanted an excuse to post some Lensa selfies.

These are really weird. I recognize them as me or a version of me, but I have no memory of them being taken. It's surreal, out-of-body. Uncanny Valley.

While we played checkers, we thought billionaires played 3D chess. They're playing the same game on a fancier board.

Every medium has nuances and norms. Texting is authentic and casual. A smaller circle has access, creating intimacy and immediacy. Most people read all their texts, but not all their email and mail. Many of us no longer listen to our voicemails, and calling your kids ages you.

Live interviews and testimony under oath inspire real moments, rare in a world where communications departments sanitize everything powerful people say. When (some of) Elon's text messages became public in Twitter v. Musk, we got a glimpse into tech power. It's bowels.

These texts illuminate the tech community's upper caste.

Checkers, Not Chess

Elon texts with Larry Ellison, Joe Rogan, Sam Bankman-Fried, Satya Nadella, and Jack Dorsey. They reveal astounding logic, prose, and discourse. The world's richest man and his followers are unsophisticated, obtuse, and petty. Possibly. While we played checkers, we thought billionaires played 3D chess. They're playing the same game on a fancier board.

They fumble with their computers.

They lean on others to get jobs for their kids (no surprise).

No matter how rich, they always could use more (money).

Differences A social hierarchy exists. Among this circle, the currency of deference is... currency. Money increases sycophantry. Oculus and Elon's "friends'" texts induce nausea.

Autocorrect frustrates everyone.

Elon doesn't stand out to me in these texts; he comes off mostly OK in my view. It’s the people around him. It seems our idolatry of innovators has infected the uber-wealthy, giving them an uncontrollable urge to kill the cool kid for a seat at his cafeteria table. "I'd grenade for you." If someone says this and they're not fighting you, they're a fan, not a friend.

Many powerful people are undone by their fake friends. Facilitators, not well-wishers. When Elon-Twitter started, I wrote about power. Unchecked power is intoxicating. This is a scientific fact, not a thesis. Power causes us to downplay risk, magnify rewards, and act on instincts more quickly. You lose self-control and must rely on others.

You'd hope the world's richest person has advisers who push back when necessary (i.e., not yes men). Elon's reckless, childish behavior and these texts show there is no truth-teller. I found just one pushback in the 151-page document. It came from Twitter CEO Parag Agrawal, who, in response to Elon’s unhelpful “Is Twitter dying?” tweet, let Elon know what he thought: It was unhelpful. Elon’s response? A childish, terse insult.

Scale

The texts are mostly unremarkable. There are some, however, that do remind us the (super-)rich are different. Specifically, the discussions of possible equity investments from crypto-billionaire Sam Bankman-Fried (“Does he have huge amounts of money?”) and this exchange with Larry Ellison:

Ellison, who co-founded $175 billion Oracle, is wealthy. Less clear is whether he can text a billion dollars. Who hasn't been texted $1 billion? Ellison offered 8,000 times the median American's net worth, enough to buy 3,000 Ferraris or the Chicago Blackhawks. It's a bedrock principle of capitalism to have incredibly successful people who are exponentially wealthier than the rest of us. It creates an incentive structure that inspires productivity and prosperity. When people offer billions over text to help a billionaire's vanity project in a country where 1 in 5 children are food insecure, isn't America messed up?

Elon's Morgan Stanley banker, Michael Grimes, tells him that Web3 ventures investor Bankman-Fried can invest $5 billion in the deal: “could do $5bn if everything vision lock... Believes in your mission." The message bothers Elon. In Elon's world, $5 billion doesn't warrant a worded response. $5 billion is more than many small nations' GDP, twice the SEC budget, and five times the NRC budget.

If income inequality worries you after reading this, trust your gut.

Billionaires aren't like the rich.

As an entrepreneur, academic, and investor, I've met modest-income people, rich people, and billionaires. Rich people seem different to me. They're smarter and harder working than most Americans. Monty Burns from The Simpsons is a cartoon about rich people. Rich people have character and know how to make friends. Success requires supporters.

I've never noticed a talent or intelligence gap between wealthy and ultra-wealthy people. Conflating talent and luck infects the tech elite. Timing is more important than incremental intelligence when going from millions to hundreds of millions or billions. Proof? Elon's texting. Any man who electrifies the auto industry and lands two rockets on barges is a genius. His mega-billions come from a well-regulated capital market, enforceable contracts, thousands of workers, and billions of dollars in government subsidies, including a $465 million DOE loan that allowed Tesla to produce the Model S. So, is Mr. Musk a genius or an impressive man in a unique time and place?

The Point

Elon's texts taught us more? He can't "fix" Twitter. For two weeks in April, he was all in on blockchain Twitter, brainstorming Dogecoin payments for tweets with his brother — i.e., paid speech — while telling Twitter's board he was going to make a hostile tender offer. Kimbal approved. By May, he was over crypto and "laborious blockchain debates." (Mood.)

Elon asked the Twitter CEO for "an update from the Twitter engineering team" No record shows if he got the meeting. It doesn't "fix" Twitter either. And this is Elon's problem. He's a grown-up child with all the toys and no boundaries. His yes-men encourage his most facile thoughts, and shitposts and errant behavior diminish his genius and ours.

Post-Apocalyptic

The universe's titans have a sense of humor.

Every day, we must ask: Who keeps me real? Who will disagree with me? Who will save me from my psychosis, which has brought down so many successful people? Elon Musk doesn't need anyone to jump on a grenade for him; he needs to stop throwing them because one will explode in his hand.

My kids redesigned the internet because it lacks inventiveness.

Internet today is bland. Everything is generic: fonts, layouts, pages, and visual language. Microtypography is messy.

Web design today seems dictated by technical and ideological constraints rather than creativity and ideas. Text and graphics are in containers on every page. All design is assumed.

Ironically, web technologies can design a lot. We can execute most designs. We make shocking, evocative websites. Experimental typography, generating graphics, and interactive experiences are possible.

Even designer websites use containers in containers. Dribbble and Behance, the two most popular creative websites, are boring. Lead image.

How did this happen?

Several reasons. WordPress and other blogging platforms use templates. These frameworks build web pages by combining graphics, headlines, body content, and videos. Not designs, templates. These rules combine related data types. These platforms don't let users customize pages beyond the template. You filled the template.

Templates are content-neutral. Thus, the issue.

Form should reflect and shape content, which is a design principle. Separating them produces content containers. Templates have no design value.

One of the fundamental principles of design is a deep and meaningful connection between form and content.

Web design lacks imagination for many reasons. Most are pragmatic and economic. Page design takes time. Large websites lack the resources to create a page from scratch due to the speed of internet news and the frequency of new items. HTML, JavaScript, and CSS continue to challenge web designers. Web design can't match desktop publishing's straightforward operations.

Designers may also be lazy. Mobile-first, generic, framework-driven development tends to ignore web page visual and contextual integrity.

How can we overcome this? How might expressive and avant-garde websites look today?

Rediscovering the past helps design the future.

'90s-era web design

At the University of the Arts Bremen's research and development group, I created my first website 23 years ago. Web design was trendy. Young web. Pages inspired me.

We struggled with HTML in the mid-1990s. Arial, Times, and Verdana were the only web-safe fonts. Anything exciting required table layouts, monospaced fonts, or GIFs. HTML was originally content-driven, thus we had to work against it to create a page.

Experimental typography was booming. Designers challenged the established quo from Jan Tschichold's Die Neue Typographie in the twenties to April Greiman's computer-driven layouts in the eighties. By the mid-1990s, an uncommon confluence of technological and cultural breakthroughs enabled radical graphic design. Irma Boom, David Carson, Paula Scher, Neville Brody, and others showed it.

Early web pages were dull compared to graphic design's aesthetic explosion. The Web Design Museum shows this.

Nobody knew how to conduct browser-based graphic design. Web page design was undefined. No standards. No CMS (nearly), CSS, JS, video, animation.

Now is as good a time as any to challenge the internet’s visual conformity.

In 2018, everything is browser-based. Massive layouts to micro-typography, animation, and video. How do we use these great possibilities? Containerized containers. JavaScript-contaminated mobile-first pages. Visually uniform templates. Web design 23 years later would disappoint my younger self.

Our imagination, not technology, restricts web design. We're too conformist to aesthetics, economics, and expectations.

Crisis generates opportunity. Challenge online visual conformity now. I'm too old and bourgeois to develop a radical, experimental, and cutting-edge website. I can ask my students.

I taught web design at the Potsdam Interface Design Programme in 2017. Each team has to redesign a website. Create expressive, inventive visual experiences on the browser. Create with contemporary web technologies. Avoid usability, readability, and flexibility concerns. Act. Ignore Erwartungskonformität.

The class outcome pleased me. This overview page shows all results. Four diverse projects address the challenge.

1. ZKM by Frederic Haase and Jonas Köpfer

Frederic and Jonas began their experiments on the ZKM website. The ZKM is Germany's leading media art exhibition location, but its website remains conventional. It's useful but not avant-garde like the shows' art.

Frederic and Jonas designed the ZKM site's concept, aesthetic language, and technical configuration to reflect the museum's progressive approach. A generative design engine generates new layouts for each page load.

ZKM redesign.

2. Streem by Daria Thies, Bela Kurek, and Lucas Vogel

Street art magazine Streem. It promotes new artists and societal topics. Streem includes artwork, painting, photography, design, writing, and journalism. Daria, Bela, and Lucas used these influences to develop a conceptual metropolis. They designed four neighborhoods to reflect magazine sections for their prototype. For a legible city, they use powerful illustrative styles and spatial typography.

Streem makeover.

3. Medium by Amelie Kirchmeyer and Fabian Schultz

Amelie and Fabian structured. Instead of developing a form for a tale, they dissolved a web page into semantic, syntactical, and statistical aspects. HTML's flexibility was their goal. They broke Medium posts into experimental typographic space.

Medium revamp.

4. Hacker News by Fabian Dinklage and Florian Zia

Florian and Fabian made Hacker News interactive. The social networking site aggregates computer science and IT news. Its voting and debate features are extensive despite its simple style. Fabian and Florian transformed the structure into a typographic timeline and network area. News and comments sequence and connect the visuals. To read Hacker News, they connected their design to the API. Hacker News makeover.

Communication is not legibility, said Carson. Apply this to web design today. Modern websites must be legible, usable, responsive, and accessible. They shouldn't limit its visual palette. Visual and human-centered design are not stereotypes.

I want radical, generative, evocative, insightful, adequate, content-specific, and intelligent site design. I want to rediscover web design experimentation. More surprises please. I hope the web will appear different in 23 years.

Update: this essay has sparked a lively discussion! I wrote a brief response to the debate's most common points: Creativity vs. Usability

I started smoking in 2002 at IIT BHU. Most of us thought it was enjoyable at first. I didn't realize the cost later.

In 2005, during my final semester, I lost my father. Suddenly, I felt more accountable for my mother and myself.

I quit before starting my first job in Bangalore. I didn't see any smoking friends in my hometown for 2 months before moving to Bangalore.

For the next 5-6 years, I had no regimen and smoked only when drinking.

Due to personal concerns, I started smoking again after my 2011 marriage. Now smoking was a constant guilty pleasure.

I smoked 3-4 cigarettes a day, but never in front of my family or on weekends. I used to excuse this with pride! First office ritual: smoking. Even with guilt, I couldn't stop this time because of personal concerns.

After 8-9 years, in mid 2019, a personal development program solved all my problems. I felt complete in myself. After this, I just needed one cigarette each day.

The hardest thing was leaving this final cigarette behind, even though I didn't want it.

James Clear's Atomic Habits was published last year. I'd only read 2-3 non-tech books before reading this one in August 2021. I knew everything but couldn't use it.

In April 2022, I realized the compounding effect of a bad habit thanks to my subconscious mind. 1 cigarette per day (excluding weekends) equals 240 = 24 packs per year, which is a lot. No matter how much I did, it felt negative.

Then I applied the 2nd principle of this book, identifying the trigger. I tried to identify all the major triggers of smoking. I found social drinking is one of them & If I am able to control it during that time, I can easily control it in other situations as well. Going further whenever I drank, I was pre-determined to ignore the craving at any cost. Believe me, it was very hard initially but gradually this craving started fading away even with drinks.

I've been smoke-free for 3 months. Now I know a bad habit's effects. After realizing the power of habits, I'm developing other good habits which I ignored all my life.