Data Pipelines With Snowflake And Streamlit

baladia

Member

Amateur

Nov 14, 2024

Data Pipelines With Snowflake And Streamlit
Published 9/2024
MP4 | Video: h264, 1920x1080 | Audio: AAC, 44.1 KHz
Language: English | Size: 1.97 GB | Duration: 5h 18m

Using Snowflake to data engineer Kaggle and Google Trends data with Python procedures and tasks

What you'll learn

Setup Snowflake and AWS Accounts

Work with Kaggle and SerpAPI

Download and manipulate data with Jupyter Notebooks on VS Code

Work with External Access Integration and Storage Integration on Snowflake

Create Snowflake Python based procedures

Create Snowflake tasks

Create Streamlit apps inside of Snowflake

Requirements

Proficient knowledge on SQL and basic knowledge on Snowflake database

Basic knowledge on data modeling and engineering

Proficient Python knowledge

Description

This course focuses on building a data engineering pipeline that integrates multiple data sources, including Kaggle datasets and Google Trends data (fetched via SerpAPI), to analyze the relationship between Netflix show releases and the popularity of actors. You'll learn to gather and combine data on Netflix actors and their trends on Google, particularly in the weeks following a show's release.You will use Kaggle as a source for the Netflix shows and actors dataset and Google Trends (accessed via SerpAPI) to fetch real-time search data for the actors. This data will be stored and processed within the Snowflake database, leveraging its cloud-native architecture for optimal scalability and performance.Technical Stack Overview:Snowflake Database: The central repository for storing and querying data.Streamlit in Snowflake: A web app framework to visualize the data directly inside Snowflake.AWS S3: For data storage and retrieval, particularly for intermediate datasets.Snowflake Python Procedures: Automating data manipulation and pipeline processes.Snowflake External Access & Storage Integrations: Managing secure access to external services and storage.By the end of the course, you'll have a fully functional data pipeline that processes and combines streaming data, cloud storage, and APIs for trend analysis, visualized through an interactive Streamlit app within Snowflake.

To view the content, you need to Sign In or Register.

Click to read more...

Data Pipelines With Snowflake And Streamlit

Currently reading: Data Pipelines With Snowflake And Streamlit

baladia

"S'all Good, Man."​

"Perfection Is The Enemy Of Perfectly Adequate."​

"Money Is The Point!"​

"I Travel In Worlds You Can't Even Imagine."​

"Say Nothing, You Understand? Get A Lawyer!"​

“Confidence is good. Facts on your side, better.” ​

“Facts are facts.”​

“Sometimes the good guys win.”​

“I’m not good at building shit, you know? I’m excellent at tearing it down.”​

“Money is not beside the point… Money is the point.”​

“Whoa, whoa. Hold up. What the hell happened to you? I get it, the first rule of Fight Club, right?”​

“A good magician never reveals his secrets.”​

“Got to look successful to be successful.”​

“The lesson is, if you’re gonna be a criminal, do your homework.” ​

“If I had to do it all over again, I would maybe do some things differently. I just thought you should know that.”​

“Some men aren't looking for anything logical. They can't be bought, bullied, reasoned or negotiated with. Some men just want to watch the world burn.”​

"Ernest Hemingway once wrote, "The world is a fine place and worth fighting for." I agree with the second part."​

“There’s no better way to destroy someone’s life than to uncover their secrets.”​

“Hackers are breaking the systems for profit. Before, it was about intellectual curiosity and pursuit of knowledge and thrill, and now hacking is big business.”​

“Hackers often describe what they do as playfully creative problem-solving.”​

“Computer hackers do not need to know each other’s real names, or even live on the same continent, to steal millions in mere hours."​

“While many hackers have the knowledge, skills, and tools to attack computer systems, they generally lack the motivation to cause violence or severe economic or social harm.”​

“Very smart people are often tricked by hackers, by phishing. I don’t exclude myself from that. It’s about being smarter than a hacker. Not about being smart.”​

“At the end of the day, my goal was to be the best hacker.”​

“Humiliation is the favorite currency of the hacker.”​

“The hacker didn’t succeed through sophistication. Rather he poked at obvious places, trying to enter through unlocked doors. Persistence, not wizardry, let him through.”​

"Rules. Without Them We Live With The Animals.”​

“Consider This A Professional Courtesy.”​

"I've Lived My Life My Way, And I'll Die My Way."​

"You stabbed the devil in the back, and forced him back into the life that he had just left."​

"You Want A War, Or Do You Want To Just Give Me A Gun?"​

"Leave one wolf alive and the sheep are never safe."​

"When you play the game of thrones, you win or you die. There is no middle ground."​

"It's not easy to see something that’s never been before: A good world."​

"I believe in second chances. I don't believe in third chances."​

"If you only trust the people you grew up with, you won't make many allies."​

"A man with no motive is a man no one suspects. Always keep your foes confused: If they don't know who you are, what you want—they can't know what you plan to do next."​

"Never forget what you are, the rest of the world will not. Wear it like armor and it can never be used to hurt you."​

"I try to know as many people as I can. You never know which one you'll need."​

"It's hard to put a leash on a dog once you've put a crown on its head."​

“Everything before the word ‘but’ is horseshit.”​

“A lion doesn’t concern himself with the opinions of a sheep.”​

“Nothing FUCKS you harder than time.”​

“You pray for rain, you gotta deal with the mud too. That’s a part of it.”​

“I’d be more frightened by not using whatever abilities I’d been given.”​

“Luck is where opportunity meets preparation.”​

“If you have an enemy, then learn and know your enemy, don’t just be mad at him or her.”​

“Every failed experiment is one step closer to success.”​

When you work on a computer your hands travel 20 kilometres a day!​

Fugaku supercomputer is the world’s fastest computer. The $1-billion supercomputer has 7,630,848 cores, requires 29,899 kilowatts of electricity, and can execute 442,010 teraFLOPs.​

“Every day, about 317 million new viruses are discovered.​

“Microsoft’s founder, the infamous Bill Gates, was actually a college dropout."​

Did you know?​

“On average, a human blinks 20 times per minute, but using a computer reduces it to 7."​

Did you know?​

“The most common password for a computer and social media platforms is 123456."​

Did you know?​

“There are eight varieties of computers: mainframe, supercomputer, workstation, personal computer, Apple Macintosh, laptop, tablet, and smartphone."​

Did you know?​

“Linux leads the industry as it is used by Google, Facebook, Twitter, and Amazon."​

Did you know?​

“NASA computers were hijacked by a 15-year-old, resulting in a 21-day halt."​

Did you know?​

“You may heat a room with Gaming PCs more effectively than a heater."​

Did you know?​

“Physical money accounts for just around 10% of global cash, while the rest is stored on computers."​

Did you know?​

“YouTube actually started as a dating website." (Oh crap xD)​

Did you know?​

“Before they could progress as stable brands, Microsoft, HP, and Apple began manufacturing computers in their Garages."​

Did you know?​

“For every 12 million email spams, only one gets a reply."​

Did you know?​

“Banks and other corporate giants hire white hats or “good hackers” to help fix security issues and prevent system infiltration."​

Did you know?​

“If Earth stopped rotating for 1 second, everyone would die."​

Did you know?​

Currently reading:
Data Pipelines With Snowflake And Streamlit

"S'all Good, Man."

"Perfection Is The Enemy Of Perfectly Adequate."

"Money Is The Point!"

"I Travel In Worlds You Can't Even Imagine."

"Say Nothing, You Understand? Get A Lawyer!"

“Confidence is good. Facts on your side, better.”

“Facts are facts.”

“Sometimes the good guys win.”

“I’m not good at building shit, you know? I’m excellent at tearing it down.”

“Money is not beside the point… Money is the point.”

“Whoa, whoa. Hold up. What the hell happened to you? I get it, the first rule of Fight Club, right?”

“A good magician never reveals his secrets.”

“Got to look successful to be successful.”

“The lesson is, if you’re gonna be a criminal, do your homework.”

“If I had to do it all over again, I would maybe do some things differently. I just thought you should know that.”

“Some men aren't looking for anything logical. They can't be bought, bullied, reasoned or negotiated with. Some men just want to watch the world burn.”

"Ernest Hemingway once wrote, "The world is a fine place and worth fighting for." I agree with the second part."

“There’s no better way to destroy someone’s life than to uncover their secrets.”

“Hackers are breaking the systems for profit. Before, it was about intellectual curiosity and pursuit of knowledge and thrill, and now hacking is big business.”

“Hackers often describe what they do as playfully creative problem-solving.”

“Computer hackers do not need to know each other’s real names, or even live on the same continent, to steal millions in mere hours."

“While many hackers have the knowledge, skills, and tools to attack computer systems, they generally lack the motivation to cause violence or severe economic or social harm.”

“Very smart people are often tricked by hackers, by phishing. I don’t exclude myself from that. It’s about being smarter than a hacker. Not about being smart.”

“At the end of the day, my goal was to be the best hacker.”

“Humiliation is the favorite currency of the hacker.”

“The hacker didn’t succeed through sophistication. Rather he poked at obvious places, trying to enter through unlocked doors. Persistence, not wizardry, let him through.”

"Rules. Without Them We Live With The Animals.”

“Consider This A Professional Courtesy.”

"I've Lived My Life My Way, And I'll Die My Way."

"You stabbed the devil in the back, and forced him back into the life that he had just left."

"You Want A War, Or Do You Want To Just Give Me A Gun?"

"Leave one wolf alive and the sheep are never safe."

"When you play the game of thrones, you win or you die. There is no middle ground."

"It's not easy to see something that’s never been before: A good world."

"I believe in second chances. I don't believe in third chances."

"If you only trust the people you grew up with, you won't make many allies."

"A man with no motive is a man no one suspects. Always keep your foes confused: If they don't know who you are, what you want—they can't know what you plan to do next."

"Never forget what you are, the rest of the world will not. Wear it like armor and it can never be used to hurt you."

"I try to know as many people as I can. You never know which one you'll need."

"It's hard to put a leash on a dog once you've put a crown on its head."

“Everything before the word ‘but’ is horseshit.”

“A lion doesn’t concern himself with the opinions of a sheep.”

“Nothing FUCKS you harder than time.”

“You pray for rain, you gotta deal with the mud too. That’s a part of it.”

“I’d be more frightened by not using whatever abilities I’d been given.”

“Luck is where opportunity meets preparation.”

“If you have an enemy, then learn and know your enemy, don’t just be mad at him or her.”

“Every failed experiment is one step closer to success.”

When you work on a computer your hands travel 20 kilometres a day!

Fugaku supercomputer is the world’s fastest computer. The $1-billion supercomputer has 7,630,848 cores, requires 29,899 kilowatts of electricity, and can execute 442,010 teraFLOPs.

“Every day, about 317 million new viruses are discovered.

“Microsoft’s founder, the infamous Bill Gates, was actually a college dropout."

Did you know?

“On average, a human blinks 20 times per minute, but using a computer reduces it to 7."

Did you know?

“The most common password for a computer and social media platforms is 123456."

Did you know?

“There are eight varieties of computers: mainframe, supercomputer, workstation, personal computer, Apple Macintosh, laptop, tablet, and smartphone."

Did you know?

“Linux leads the industry as it is used by Google, Facebook, Twitter, and Amazon."

Did you know?

“NASA computers were hijacked by a 15-year-old, resulting in a 21-day halt."

Did you know?

“You may heat a room with Gaming PCs more effectively than a heater."

Did you know?

“Physical money accounts for just around 10% of global cash, while the rest is stored on computers."

Did you know?

“YouTube actually started as a dating website." (Oh crap xD)

Did you know?

“Before they could progress as stable brands, Microsoft, HP, and Apple began manufacturing computers in their Garages."

Did you know?

“For every 12 million email spams, only one gets a reply."

Did you know?

“Banks and other corporate giants hire white hats or “good hackers” to help fix security issues and prevent system infiltration."

Did you know?

“If Earth stopped rotating for 1 second, everyone would die."

Did you know?

“If someone made a sound of 1100db or larger a black hole would form sucking in our whole solar system."

“People shouldn't be afraid of their government. Governments should be afraid of their people.”