Fully Proofread Edition

Author's Note:
This book consists of twenty chapters, divided into four parts. The full text is approximately 215,000 words. The following is a meticulously edited version, which has standardized the format, corrected typos, optimized the fluency of the sentences, and maintained the original narrative style and emotional tone.

📑 Table of Contents (Publisher-Ready)

Preface

Part I — Lost (Reality Shock)

The 127th Rejection Letter
Winter in Toronto
Does Data Not Lie?
My First Job
The First Day of Automation

Part II — Awakening (Skills and Cognition)

The Fear of Being Replaced
SQL in the Dead of Night
The Invisible System
The First Collapse
The World of Engineers

Part III — Restructuring (Entering the Core Layer)

Data Pipelines
Behind AI
Data Quality Wars
The Architecture Mindset
From Analyst to Engineer

Part IV — Evolution (System Thinking and Identity)

Scaling Systems
Ownership and Responsibility
The Cost of Failure
Designing for the Unknown
The Engineer’s Mind

✍️ Preface (Publisher Style)

In recent years, “data” has become one of the most celebrated concepts in technology and business. Terms like data science, AI, and machine learning dominate headlines, shaping how we imagine the future. Yet beneath this visible layer lies a largely overlooked reality: most organizations struggle not with advanced models, but with the fundamental challenge of making data usable.

This book was written to explore that hidden world.

The Journey of a Data Engineer is not a traditional technical manual, nor is it purely a work of fiction. It sits at the intersection of narrative and reality, using storytelling to reveal the structures, tensions, and transformations within modern data work.

Through the experiences of Lin Chuan, readers will encounter questions that extend beyond technology:

Why does education often fail to prepare people for real-world systems?
What does it mean to create value in a data-driven economy?
How do individuals adapt when automation begins to replace their work?
And perhaps most importantly: what does the world actually need from us?

This story is grounded in real practices—SQL, data pipelines, ETL processes, system failures—but its core is human. It is about uncertainty, reinvention, and the gradual construction of competence in a world that rarely provides clear answers.

For students, it offers a bridge between theory and practice.
For professionals, it reflects the hidden struggles behind everyday systems.
For general readers, it reveals the invisible infrastructure that quietly supports modern life.

In the end, this book argues a simple but profound idea:

The people who shape the future of data are not those who merely analyze it, but those who make it possible.

Part 1: Lost ( Reality Shock )

Chapter 1: The 127th Rejection Letter

Lin Chuan stared at the screen. The cursor blinked rhythmically across the email interface, like a heartbeat .

He no longer needs to open the email to know what it says .

“We regret to inform you…”

Letter 127 .

He hovered his mouse over " Delete , " paused for two seconds, then moved it away. He didn't delete it. For some reason, he felt these emails should remain, like evidence. Proof that he had tried. Proof that the system actually existed .

Outside the window was Toronto in winter. The greyish-white sky resembled an uncolored chart — calm, precise, and emotionless . The November snow hadn't really started falling yet, but the wind was already cold. His rented apartment was on the second floor, and the radiators rattled like an old man coughing .

He suddenly thought of a question: What would it be like to describe himself using data ?

He opened a new document and typed a few lines :

yaml

age : 27

Education : Master of Statistics (University of Toronto)

Total number of resumes submitted : 127

Received an interview : 4

Entering the final interview : 1

Offer : 0

Unemployment days : 187

Bank balance : 2,847 Canadian dollars

A standard failure sample .

He gave a wry smile .

college , a professor said something in the first class that he still remembers :

Statistics is the language of understanding the world .

But now, he can't even explain his own life .

He opened LinkedIn . The screen displayed another world .

· "I'm thrilled to announce that I've joined Microsoft as a Data Scientist!"

· "After an amazing journey at RBC, I'm starting a new chapter…"

· “Grateful for the opportunity to work with an incredible team.”

Each post is like a carefully selected data point — showing only " successes , " not " failures . " No one writes on LinkedIn , " Got rejected again today. " No one writes on LinkedIn , " I haven't had any income for three months. "

He suddenly realized : this is not data, this is filtered reality .

He clicked on a job posting :

Data Analyst – Entry Level

Require :

· 3+ years of work experience

· Proficient in Python , SQL , and Tableau

· Familiar with machine learning and statistical modeling

· Has cloud computing experience ( AWS/Azure )

· Has business analysis experience

He stared at the line " Entry Level" as if it were a joke .

Entry Level requires three years of experience. What kind of logic is that? He recalled a joke: You need work experience to find a job, but you need work to gain experience. The joke is funny. Until it becomes your life .

My phone vibrated .

a message from a classmate .

" You still haven't found a job? I got into RBC , doing risk analytics . The salary is decent, starting at 70k . "

The person who sent the message was Chen Haoran. They were classmates in graduate school. His grades weren't as good as Lin Chuan's — at least that's what Lin Chuan thought. In the final exam, Lin Chuan scored 92 , while Haoran scored 84. For their graduation projects, Lin Chuan's thesis was praised by the professor as " in-depth , " while Haoran's was said to be " practical but lacking in theoretical solidity . "

Ke Haoran found a job .

Lin Chuan stared at the message without replying .

Of course he knows about " risk analytics ." He's even written papers on the subject. He's used SAS to build credit scoring models and R to predict default probabilities. He knows about p- values, AUC , and how to tune the parameters of random forests .

But the reality is—

Being able to solve problems ≠ being able to work

Knowing formulas ≠ Knowing how to make money

He stood up and walked to the window .

The street downstairs was nearly deserted. Toronto 's winter slows everything down. Snow, trampled into gray, piled up on the side of the road. An elderly man pushed a shopping cart past, its cart overflowing with plastic bags. The sign of the Vietnamese pho shop across the street shone with a yellow light, its windows fogged up .

He suddenly had a strange feeling :

The world doesn't need him .

It's not directed at him personally. It's not that he's not good enough. It's just—

This system doesn't need someone who 's only good at statistics .

He returned to his computer .

I opened a job posting. Not the job description, but the comments section below. People were discussing the current state of the industry :

" The company doesn't lack analysts; what it lacks are people who can effectively utilize data. Simply knowing how to run models isn't enough ; the company needs people who can solve real-world problems. "

He stared at that sentence for a long time .

" Make use of it . "

These three words slowly unfolded in his mind, like a drop of ink falling into water .

What does that mean? What does " using " mean ? Isn't what he learned about using data methods ? Regression analysis, hypothesis testing, time series analysis — aren't these all " using data " ?

But ……

He suddenly asked himself a question :

" If a company gave you a bunch of raw data and asked you to help them earn 10% more money, would you know where to start? "

He didn't know .

He knows how to perform regression , but he doesn't know which business problem requires it. He knows how to perform hypothesis testing, but he doesn't know what hypothesis his boss wants to test. He can write elegant mathematical derivations, but if you ask him the simplest question— " How does the company make money from data ? "— he falls silent .

He began to recall .

During his four years of university, he learned :

· Probability Theory and Mathematical Statistics

· Regression analysis

· Hypothesis testing

· Time series analysis

· Multivariate statistical analysis

· Introduction to Machine Learning

· Bayesian statistics

· Statistical calculation

He can prove a model converges. He can explain the meaning of the p- value. He can write elegant mathematical derivations. He can even discuss the convergence speed of Bootstrap with professors .

But he was never taught a single course :

· Where does the data come from?

· Why is the data dirty?

· connect two databases ?

· How to handle missing values (not by filling with the mean, but in a production environment )?

· How to automate an analysis process daily?

· How to communicate requirements with product managers

· How do you measure whether an analysis is valuable?

He studied " methods of describing the world " .

What the company needs is " the ability to change the world " .

That night, he didn't send out any more resumes .

He opened a new tab .

In the search box, he slowly typed a few words :

Data Engineer What is it ?

He pressed Enter .

At that moment, he had no idea that this decision would change everything for him .

The search results are in. The title of the first article is :

“Data Engineer: The Most Underrated Job in Tech”

He clicked on it .

The first paragraph of the article :

" Data engineers are the people who build and maintain data infrastructure. They make data analyzable, usable, and trustworthy. Without data engineers, data scientists would be staring blankly at empty databases. "

He read it three times .

" Infrastructure " .

This term was somewhat unfamiliar to him. He was used to " model, " " distribution, " and " hypothesis, " rather than " system ," " pipeline, " and " architecture . "

He opened another article. This time it was a blog post with a very straightforward title :

Why are data engineers more sought-after than data scientists ?

There was a passage in it that he read repeatedly many times afterward :

" Most companies' problem isn't a lack of good models, but a lack of good data. The data is messy, fragmented, and unreliable. The job of a data engineer is to transform this messy data into clean, reliable, and usable assets. A good data engineer is harder to find and more valuable than a good data scientist. "

He leaned back in his chair .

A string in my mind was plucked .

Late that night, he made a decision .

It's not about " changing careers . " It's not about " giving up statistics . " It's about—

what it was like when the data was actually being used .

He opened a learning website. He registered. He selected a course .

First course: SQL for Data Analysis .

The course description states :

“No prior experience required. Learn how to query databases and extract insights.”

"From scratch."

Start from scratch .

He clicked on it .

outside picked up. The radiator clicked twice more. His screen lit up, illuminating the table he'd looked at hundreds of times — piled with printed rejection letters, an empty coffee cup, and a half-open copy of " Introduction to Statistical Learning . "

He closed it and pushed it aside .

The space has been freed up .

Chapter Two : Winter in Toronto

The next morning, Lin Chuan sat down in a coffee shop .

The shop wasn't big, but it was warm. A thin layer of fog clung to the glass windows, blurring the outside world into a grayish-white blur. Inside, a jazz piece he couldn't name was playing, the piano notes falling like raindrops .

He ordered the cheapest coffee — a medium Americano, for 2.25 Canadian dollars .

It wasn't because I liked it, but because of the budget .

$ 2,847 in his bank account . Rent was $1,200 a month . Phone bill, internet, insurance, food — he calculated that if he didn't spend recklessly, he could last another two and a half months .

Two and a half months .

He turned on his computer. On the screen was what he had found the night before :

Data Engineer

Responsible for building and maintaining the data infrastructure so that data can be analyzed and used .

Typical skills: Python , SQL , data warehousing, ETL , cloud platforms ( AWS/GCP/Azure ), Spark , Airflow .

* Average Salary (Canada): 85,000 - 120,000 CAD/ year *

He stared at that salary range for a long time .

It wasn't because it was high. It was because the " data analyst " positions he had applied for previously only averaged between 55k and 70k .

It's not about money. It's about " value " .

The two people at the next table were chatting. Their voices weren't loud, but the coffee shop was so small that he could hear them clearly .

" Our biggest problem right now isn't the model, it's the data. Running reports takes two hours every day because the pipeline crashes constantly. "

" Yes, the customer churn prediction model from last week was completely unreliable. We later discovered that the data was duplicated in the feature engineering layer. "

" You're still using Python to run this? "

" I'm using Spark , but the data skew has always been severe, and I haven't been able to optimize it properly. "

Lin Chuan subconsciously raised his head .

Pipeline . Spark . Data skew .

He had just seen these words online. They were like the language of another world — not more advanced , but on a completely different dimension .

He suddenly realized that they were talking about a world he had never learned about .

He looked back at the screen. Scrolling down the blog post, he had highlighted a passage with his eyes :

"80% of data science time isn't spent on modeling, but on data processing. Cleaning, integrating, debugging, and optimizing — these are the daily routines of data work. If you only want to build models, you'll be disappointed. If you want to solve problems, you'll fall in love with data engineering. "

He frowned .

This statement is completely contrary to his understanding over the past seven years .

Everything at school is clean :

· CSV file prepared by the professor.

· Missing values have been handled .

· The format is uniform .

· The model is the core.

· The result is certain .

But the real world seems to be :

· The data is chaotic, scattered, and contradictory .

· The system is complex, fragile, and opaque .

· The problem is vague, variable, and has no standard answer .

· The results are probabilistic, open to questioning, and require explanation .

He suddenly felt a little uneasy .

If this is true — if he really didn't spend 80% of his time modeling — then what exactly has he been learning over the past few years ?

It's not that what he learned was useless. Rather, what he learned was just the tip of the iceberg. Or even just a small piece at the very tip of the iceberg .

He opened a job posting. Position: Data Engineer .

He was asked to read through each item one by one :

Skill	His level of mastery
Python	I know how to use pandas , but I don't know how to write classes , generators, or decorators .
SQL	I know basic SELECT statements , but not window functions, CTEs , or query optimization.
Data warehouse	I don't understand at all
cloud platform	I've heard of it, but never used it.
ETL	I've heard of it, but I don't know the specifics of how it's done.
Airflow	This is the first time I've heard of it .
Spark	I've heard of it, but never used it.

He was unfamiliar with almost none of them .

He leaned back in his chair, staring at the ceiling. There was a lamp there, its shade askew, revealing the energy-saving bulb inside. The light was a little glaring .

For the first time, he sensed a clear difference :

It's not that we didn't try hard enough , but that we were going in the wrong direction .

He kept running hard, but he was running on someone else's track. He kept improving himself, but he was improving things that the market no longer needed .

Or to be more precise : what the market needs is not the kind of " data-savvy " person he thought he was .

Outside the window, a gust of wind blew the snow up. The snowflakes were small, but dense, hitting the glass at an angle .

The crowd hurried by . Some people, bundled up in thick down jackets , looked down at their phones. Others pushed strollers, walking carefully along the sidewalk. Still others stood under the bus stop, hunching their shoulders as they waited for the bus .

Everyone has their own destination .

He was like a variable that hadn't been assigned a task .

A function that has not been called .

A piece of code that was written but will never be executed .

He sat up straight again .

Not because he had the answer. But because he realized that he had at least asked the right question .

The question isn't " How do I find a job ? "

The question is, " What does the world need me to do with data ? "

He opened a learning website .

The first line reads :

"Learn SQL from scratch."

From scratch .

Start from scratch .

He clicked on it .

On the first day, he learned SELECT and FROM .

The next day, he learned WHERE and ORDER BY .

On the third day, he encountered his first problem .

The data in the tutorial is clean . Perfect. Every row has a value, and every column has the correct formatting .

link below the tutorial : " Download sample dataset for practice."

He downloaded it . He opened it .

I was stunned .

The CSV file looks like this :

order_id	customer_name	order_date	amount	status
1	John Smith	2023-01-15	29.99	completed
2	( null )	2023/01/16	-	completed
3	John Smith	2023-01-15	29.99	NULL
4	Jane Doe	15/01/2023	100	pending
5	Jane Doe	2023-01-15	100	completed
( null )	NULL	NULL	( empty string )	duplicate

· null value

· Duplicate lines

· The formatting is messed up (the date has three different formats ).

· Time field error

· inconsistent capitalization

· data types ( there's a " -" sign in the amount field ).

· There's even one empty line .

He stared at the document as if he were looking at a deliberately vandalized crime scene .

It took him two hours to get the first query. He wanted to calculate the total order amount for each customer. But the result was wrong — duplicate orders were counted twice, an empty customer_name was treated as a single customer , and the " -" in the amount caused the entire query to fail .

He rewrote it. He ran it again. Still wrong .

He revised it again. He ran it again. Okay. But the result didn't look right .

He suddenly realized something :

Data is never the " truth " .

Data is merely “ recorded fragments ” .

The process of recording this information was fraught with errors, omissions, inconsistencies, and human mistakes .

That night, he wrote the first line of his " work notes " in his notebook :

First principle : Before analyzing, you must question the data .

Ask yourself three questions :

1. How was this data generated ?

2. Which data might be problematic ?

3. If the data is wrong, will I be able to find out ?

Little did he know that this habit — skepticism of data — would save him two years later. A mistake that almost got him fired .

Chapter 3: Does Data Not Lie ?

He started going to that coffee shop every day .

It wasn't because he liked coffee. It was because in his apartment, he couldn't help but refresh his email, check LinkedIn , and think about those rejection letters .

In a coffee shop, there is at least a sense of order, as if people are " pretending to be working " .

He set a rule for himself: learn SQL in the morning , learn Python in the afternoon , and work on small projects in the evening .

A week later, he could write SQL queries with JOIN , GROUP BY , and subqueries . He learned window functions —RANK () , ROW_NUMBER() , LAG() , and LEAD() . These were things he had never been taught in school, but he had seen online that " these are essential skills for data analysis . "

He learned very quickly. Faster than he expected .

Perhaps it's because he has a master's degree in statistics. Logic, mathematics, abstract thinking — these are universal. SQL is just another language for expressing logic .

But there was one thing he learned very slowly .

debug .

In statistics, if your code has an error, it's usually a syntax error. Just fix it .

However, in data processing, code can be without any syntax errors, run very fast, and produce completely wrong results .

For example, he wrote a query to calculate the average order amount each month. The results came back, and the numbers seemed reasonable. He almost believed it .

Then he casually checked the total number of orders and found that it was half missing .

Why? Because when he performed the join , he used INNER JOIN to filter out months without orders .

There were no errors. There were no warnings. The result was simply wrong .

This is the most terrifying aspect of data work: errors are often silent .

Two weeks later, one evening, something happened that completely changed his understanding of " data " .

He was practicing with a dataset — New York taxi data. It was publicly available online, several gigabytes of CSV files. He used Python to read a sample and did some analysis .

He found that the average travel time was 40% longer between 4 pm and 7 pm on weekdays than on weekends .

That makes sense. There's a traffic jam .

Then he drew a graph and found that the average travel time was also very long, from 11 p.m. to 2 a.m. on Saturday nights .

That makes sense. The bar closed , and everyone took taxis home .

Then he noticed an anomaly: the average travel time between 3 a.m. and 4 a.m. on Wednesdays was 60% longer than the surrounding time periods .

This doesn't make sense. It's 3 a.m. , and there aren't many cars on the road .

He checked the raw data and found that in the records for that one hour, there were hundreds of trips with a duration of " 9999 minutes "—about a week .

This is clearly a data error. It's possible the GPS signal was lost, and the system entered a default value .

If he hadn't noticed this anomaly and had gone straight to the analysis, the conclusion would have been " Wednesday morning was also very congested . "

He would report this conclusion to his boss. The boss might believe him. Then the company might make some decisions based on this erroneous data .

He suddenly felt a chill on his back .

That night, he wrote a second line of notes in his notebook :

The second principle : Data doesn't lie, but people who record data can make mistakes .

Behind every dataset lies a noisy data generation process. Your job is not to " make the data speak , " but to " make the data tell the truth . "

in telling the truth is to find out where the data is lying .

A month later, he finished learning the basics of SQL and began learning Python 's data processing library , pandas .

pandas . He used it during his graduate studies. But before that, he only used basic functions like read_csv () , dropna() , groupby() , and mean() .

He began to learn more advanced things :

· Four ways to use merge( )

· pivot_table ( )

· apply( ) and lambda functions

· The `resample( )` function for processing time series data.

· Various methods for handling missing values (not just simple Dropna )

· Performance optimization: Vectorized operations vs. loops

He did an exercise: he cleaned a 1GB log file from its raw format into a format usable for analysis .

Raw data :

· 30 fields, half of which are empty .

· The timestamp is a Unix timestamp and needs to be converted.

· IP addresses need to be resolved to city names.

· user_agent needs to be parsed into browser and operating system.

· Some lines are generated by web crawlers and need to be filtered.

He spent four hours writing 200 lines of code and finally got it running .

when he saw the cleaned data neatly displayed on the screen .

It's not the satisfaction of " I proved something ." It's not the satisfaction of " My model's accuracy has improved . "

It's the satisfaction of " I've sorted out a mess . "

Like a gardener trimming weeds. Like a cleaner tidying up a room .

It's simple. But it's real .

Chapter Four: My First Job

In the sixth week, he received an email .

It's not a rejection letter .

It's an interview invitation .

A mid-sized retail company headquartered in Toronto. Position: Junior Data Analyst .

He remembered the company. He hesitated for a moment when submitting his resume because the job description said " 1-2 years of experience , " and he had zero .

But he still submitted it. Letter # 89 .

Now, they have replied .

There are three rounds of interviews .

The first round was a phone call from HR . It was very simple: self-introduction, why I wanted to apply, and expected salary. He was honest: he had just graduated, was willing to learn, and expected 60k . HR said , " Our range is 55k to 65k , no problem. "

The second round was a technical interview. A data analyst asked him about SQL . The questions weren't difficult: three tables, write a query, and calculate several metrics. He answered quickly. Then the interviewer asked : " If you find that the customer_id values in two tables don't match, what would you do? "

He thought for a moment and said , " First , check if the data types are inconsistent; one could be a string, and the other an integer. Then check for leading or trailing spaces. Next, check for capitalization issues. If that still doesn't work, ask the business team how this ID was generated. "

The other person nodded .

The third round was an interview with the manager. It was with a man in his forties named Mark . He looked tired and spoke very quickly .

Mark asked , " You studied statistics , so why did you choose to do data analysis? "

Lin Chuan thought for a moment and said , " Because I want to see the data actually used, not just theoretical derivations. "

Mark looked at him and said something that Lin Chuan would remember for a long time :

" Statistics is the weapon, but business is the battlefield. You can spend your whole life in the arsenal, but the battlefield is where you find out if a weapon is effective. "

A week later, the offer arrived .

Annual salary of CAD 58,000 . Includes two weeks of annual leave. Basic medical insurance is provided .

Not high. But enough to survive .

He accepted .

On his first day at work, he wore his best shirt — a blue one he bought on sale at Uniqlo. He arrived at the office twenty minutes early .

The company is located on the 12th floor of a glass building in downtown Toronto. The reception is an automated sign-in machine; you enter your name, and it prints a sticker. He wears it on his chest and walks into the office area .

The workstations are open-plan. Rows of desks, each with two monitors, a keyboard, and a mouse. Some people are coding with headphones on, some are on the phone, and some are eating breakfast .

Mark showed him around :

" This is the product team, this is the operations team, this is the technology team. You sit in the analysts' group, over there by the window. "

His workstation is by the window, offering a view of the CN Tower .

Mark gave him his first task :

" Please prepare last month's sales report first. The data is in the database. Run an SQL query , compile it into an Excel file , and send it to me before 4 PM . "

Lin Chuan nodded .

This is something he is most familiar with .

He opened his database connection tool , DBeaver . He located the sales table, wrote a query, and ran it. The results came out .

Then he started adjusting the formatting: column width, color, header, summary row. He saved it as an Excel file and sent it out .

The whole process took two hours .

Mark replied : " Great, we'll start automating it next week . "

“ Automation ? ” He was stunned .

Mark said , " Yes, this report needs to be done weekly. Write a script to automate it. Generate it automatically every Monday morning and send an email automatically. "

Lin Chuan opened his mouth, wanting to say " I can't . " But he swallowed the words .

He said , " Okay, I'll look into it. "

That afternoon, he began to study automation .

He asked his colleagues , " What do you use for automation? "

My colleague said , " Python . Write a script and run it on a cron job . Or use Airflow , but we haven't used it at our company yet. "

He returned to his workstation, opened Python , and began writing a script .

· Connect to database

· Execute SQL query

· Convert the result to a DataFrame

· Generating Excel using openpyxl

· Sending emails using smtplib

He wrote line by line, debugging again and again. Connection timeouts, encoding errors, attachments being too large and rejected by the mail server — one problem after another .

At 8 p.m., he finally got through .

He watched as the script automatically connected to the database, processed the data, generated reports, and sent emails. The whole process took three minutes .

He suddenly felt a complex mix of emotions .

On the one hand, he was very happy . He had succeeded .

On the other hand, he was thinking: what he spent two hours doing today would only take three minutes in the future. So what would he do with the time saved ?

He had no idea then that the answer to that question would become a knife held to his throat three months later .

Chapter 5 : The First Day of Automation

The second Monday, at nine o'clock in the morning .

Lin Chuan went to his office and opened his email .

There is a new email .

From: Automated Reporting System no-reply@company.com

Recipients: mark.chen@company.com, lin.chuan@company.com

Subject : Weekly Sales Report – Week 10

Attachment: sales_report_week10.xlsx

He opened the attachment. The format was correct. The data was correct .

The system completed the work that took him two hours in the first week in just three minutes, and it did so while he was sleeping .

Mark walked into the office, glanced at the emails, and said , " Great , it worked. "

Then he turned and left .

Lin Chuan sat at his workstation, staring at the screen .

For the first time, he felt a clear sense of danger :

If machines can do my job, then what am I left ?

That afternoon, he took the initiative to go find Mark .

"Mark , I 'd like to ask if there's anything else I can do besides this report? "

Mark thought for a moment and said , " How 's your SQL ? "

" Well enough. "

" Then go help the tech team with some data extraction. They're overwhelmed. Go talk to Daniel ; he 'll give you some tasks. "

Daniel .

Lin Chuan had no idea that this name would change his career .

That night, he went home but didn't go straight to sleep .

He turned on his computer and searched for a question :

How to become a data engineer

the search results contained an article with the title :

“The Data Engineering Roadmap – From Zero to Hero”

He clicked on it .

The article is quite long and is divided into seven parts :

1. SQL ( Advanced )

2. Python ( Intermediate )

3. Data Warehouse and Modeling

4. ETL and Data Pipelines

5. Cloud platforms ( AWS/GCP/Azure )

6. Big data tools ( Spark , Hadoop )

7. Workflow scheduling ( Airflow , dbt )

He looked at them one by one .

Some he had heard of. Some were completely unfamiliar .

But one feeling is clear :

This road is very long .

He closed his laptop and lay down on the bed .

There was a crack in the ceiling , stretching from the light fixture to the corner of the wall. He had been staring at it for hundreds of nights .

He suddenly remembered what Mark had said earlier that day :

" Good , it ran smoothly. "

It's not " You did a good job . " It's not " You're smart . " It's not " You're valuable . "

It means " this thing is working . "

In this system, the standard for evaluating a person is not " what you have learned " , but " what you can make run " .

He turned over and closed his eyes .

Tomorrow, he is going to see Daniel .

Tomorrow, he will begin learning things he doesn't know yet .

Tomorrow is a new variable .

Part Two: Awakening (Skills and Cognition )

Chapter Six: The Fear of Being Replaced

Lin Chuan started having insomnia .

It wasn't the kind of insomnia where you " toss and turn and can't fall asleep , " but something far more terrifying: he would suddenly wake up at three in the morning, his heart racing, and only one question would keep looping in his mind—

I be replaced ?

He lay in bed, staring at the crack in the ceiling. Toronto outside was silent in the darkness, broken only occasionally by the engine of a night bus. He picked up his phone, the screen's light making him squint .

3:14 AM .

He opened his email. There was a new email .

From: Automated Reporting System

Subject : Daily Sales Summary – Auto Generated

He clicked on it. The system had automatically completed the day's sales summary at 2 a.m., generated charts, and even automatically marked " Abnormal fluctuations : Sales in the eastern region decreased by 12% , attention recommended . "

We recommend paying attention to this .

The system is giving him suggestions .

He stared at those four words, feeling as if someone had pushed him from behind .

The next morning, he arrived at the office twenty minutes earlier than usual .

Emily was already at her workstation. She had joined the company three months earlier than Lin Chuan and was doing the same data analysis work. She was a very talkative person, and on her desk sat a small cactus and a photo of her and her dog .

" Good morning, Lin Chuan , " she said without looking up , " Did you see yesterday's automated report? "

" I saw it. "

" Isn't it a little scary ? " She looked up , smiled, but there was something unsettling about her smile. " I used to spend an hour every day working on that report, and now it just comes out on its own. I'm thinking ... what should I do then? "

Lin Chuan did not answer .

He sat down at his workstation and turned on his computer. On the screen was the automation script he had written yesterday. Looking at the code, he suddenly felt that it was like a knife — a knife he had sharpened himself and then handed to the system so that it could cut away his own work time .

wrong .

It's not during working hours .

That is his value .

At 10 a.m., Mark held a small group meeting .

The meeting room was small, and last week's KPIs were written on the whiteboard . Mark stood in front of the whiteboard, holding a marker in his hand .

“ Good news , ” he said . “ Our data automation rate increased from 30% to 65% last month . That means your team spent 35% less time on repetitive reports . ”

No one applauded .

Mark continued , " This means you can free up your time to do more valuable things, such as in-depth analysis, business insights, and data modeling. Don't just be a data-collecting machine. "

He paused, then glanced at everyone .

" It's not that the company doesn't need you anymore ; the company needs you to do more difficult things. "

After the meeting , Lin Chuan returned to his workstation .

He opened a document and began making a list :

Things that will lead to my replacement :

· Daily , weekly, and monthly reports in a fixed format

· Simple data summarization and chart generation

· Standardized queries for common questions

· data quality

I won't be replaced for :

· Understanding ambiguous business requirements

· Handling unforeseen anomalies in data

· with business stakeholders and align communication strategies .

· Make analytical conclusions based on business judgment.

· Manual work involved in fixing data pipeline problems .

Looking at the list, he felt a little more at ease .

But only a little .

Because he knows that " not yet being replaced " does not mean " never being replaced " .

In the afternoon, he passed by the technical team's workstations and heard Daniel talking to another engineer .

" Is the automatic anomaly labeling feature you're using Z-score ? "

" Yes, anything exceeding two standard deviations will be highlighted in red. "

" But sales data isn't normally distributed, so using Z-score will result in many false positives. "

" Then what should we use? "

" Use IQR , or a moving average over time series. I'll revise it this afternoon. "

Lin Chuan stood to the side and listened for a while. He found that he could understand it — not because he had learned it at the company, but because he had taught himself these things in the coffee shop .

He suddenly realized a fact :

At school, he learned that " this is a normal distribution " .

In his work, what he needs is " your data is not normally distributed , so don't use Z-score" .

The former is knowledge . The latter is judgment .

Judgment comes from experience. It comes from making mistakes. It comes from someone waking you up at 3 a.m. and telling you the pipes have collapsed .

That night, he didn't go straight home .

He spent two extra hours in the office, trying to write a more complex query in the test environment — using window functions to calculate each customer's 90- day rolling purchase amount. He ran it three times, and each time it failed. On the fourth time, he discovered the problem: he was missing a field in the PARTITION BY clause .

He corrected it. He ran again. That's right .

He stared at the results on the screen and suddenly remembered something .

A week ago, Mark asked him , " Could you analyze which customers are at the highest risk of churn? "

He said at the time , " I can build a logistic regression model. "

Mark glanced at him and said , " The model is fine, but you need to tell me first , what percentage of customers haven't made a purchase in the last 90 days ? "

He didn't know. Because he had never calculated it .

Now he knows .

It's not because of the model. It's because of a window function .

He shut down his computer and walked out of the office .

Toronto was quiet at night. There were still people walking on Yonge Street , but it wasn't as crowded as during the day. A homeless man sat on a street corner with a paper cup in front of him. Lin Chuan walked past him, hesitated for a moment, and didn't stop .

He walked home. He opened the door. The apartment was cold. The radiators were clicking and creaking again .

He lay down on the bed without turning on the light .

That question is still on my mind .

I be replaced ?

He suddenly thought of an answer — not the final answer, but at least a direction :

What cannot be replaced is not a particular skill, but rather the " ability to solve unknown problems " .

If the problem is known and the answer is standardized, machines will eventually do a better job than you .

But if the problem is vague, uncertain, and requires judgment — at least for now, machines can't do that .

He rolled over .

" At least for now . "

These two words kept him awake for another half hour .

Two weeks later, Emily was called into Mark 's office .

The door was closed. Lin Chuan couldn't see what was happening inside. But when Emily came out , she didn't look well .

" What's wrong ? " Lin Chuan asked .

“ It’s nothing , ” she said . “ The company is streamlining some processes, and one of my reports has been automated. Mark told me to do more business analysis. ”

She smiled, but the smile was more forced than before .

Lin Chuan wanted to say something, but didn't know what to say .

He returned to his workstation, opened the list , and added a line under " Things for which I will not be replaced " :

· Proactively identify business issues, rather than passively responding to demands.

What he didn't know was that three months later, Emily would walk out of that door , carrying a cardboard box containing her little cactus and a picture of the dog .

He would stand by the window and watch her walk into the elevator .

Then I went back to my workstation and continued writing code .

Chapter Seven: SQL in the Dead of Night

Lin Chuan began a new rhythm of life .

During the day, he writes reports, handles data requests, and aligns his statements with those of business stakeholders. When he gets home at night, from 11 p.m. to 3 a.m., he studies things he encounters at work but doesn't know how to do .

It wasn't because he wanted to " roll up " (i.e., take advantage of the situation) .

It was because he was afraid .

He was afraid his name would be on the next list of those to be let go. He was afraid that one day Mark would say, " This position is no longer needed . " He was afraid of going back to those days of 127 rejection letters .

So he studied late at night , like digging a tunnel in the dark .

In the first week, he learned advanced SQL .

At school, he learned the basics of SQL —SELECT , FROM , WHERE , and JOIN . But he didn't know what window functions were, what CTEs were , what query execution plans were, or what index optimization was .

He opened an online course titled " Advanced SQL for Data Engineers " .

He was stunned in the first class .

Window functions: ROW_NUMBER , RANK , DENSE_RANK , LAG , LEAD , FIRST_VALUE , LAST_VALUE

He read it three times before he could roughly understand it .

Then he did a practice problem :

Identify the top three best-selling products in each product category .

What would he have done before? Probably write a subquery, use GROUP BY , and then use ROW_NUMBER ? No, he didn't even know how to do that before. He might have exported the data to Excel and then filtered it manually .

Now, he wrote a query :

SQL

WITH ranked_products AS (

SELECT

category

product_name ,

sales_amount

ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales_amount DESC) AS rank

FROM sales_table

)

SELECT * FROM ranked_products WHERE rank <= 3;

跑了。对了。

He stared at the screen, experiencing a strange sense of pleasure .

It's not the pleasure of " I've learned it . " It's the pleasure of " I can do things I couldn't do before . "

The second week, he learned query optimization .

This part was much more difficult than he had imagined .

He used to only care about one thing when writing SQL : whether the result was correct .

Now he knows that in a production environment, the result is just the starting point. If queries run too slowly, consume too many resources, and impact other tasks — you're a problem maker .

He learned a lot of new concepts :

· Execution plan : How the database executes your query

· Index : How to speed up queries

· Partitioning : How to reduce the amount of data scanned

· Data skew : Why some nodes are 100 times slower than others

· JOIN Strategies : Differences between Nested Loop , Hash Join , and Merge Join

He tried to optimize a slow query in a real-world application. It used to take 45 seconds. He added an index, changed the JOIN order, and rewrote the subquery as a CTE . It then took 3 seconds .

3 seconds .

He couldn't believe it. He ran it again. Still 3 seconds .

He leaned back in his chair and smiled .

It wasn't because of his grades. It was because for the first time, he felt that he had transformed from " someone who knows how to use SQL " into " someone who understands why SQL works the way it does . "

In the third week, he began to learn data modeling .

In this part, his background in statistics finally came in handy .

He knows what a dimension is, what a fact is, what a consistent dimension is, and what a slowly changing dimension is .

But he has never built a model in a real-world setting .

He read a classic book— " The Data Warehouse Toolkit" —written by Kimball . It was very thick. He started reading it from the beginning , one chapter a day .

Day 1: What is a data warehouse ?

Day 2: Overview of Dimensional Modeling

Day 3: Dimensional Model of Retail Industry .

Day 4: Order Fact Sheet, Product Dimension, Time Dimension, Customer Dimension .

Day 5: Six Types of Slowly Changing Dimensions .

he saw Type 2 , a thought suddenly struck him .

Type 2 SCD : When a dimension attribute changes, the original record is not overwritten, but a new record is created, with the version identified by the expiration date and the expiration date .

He considered a real-world scenario: a customer changes their address. If the update is done directly, the address for historical orders will also become the new address, leading to incorrect analysis. Using Type 2 , historical orders are associated with the old address version of the customer record, while new orders are associated with the new address .

“ Oh , ” he said aloud , “ I see . ”

He had never thought about this problem before . Because in academic datasets, customer addresses are fixed. In the real world, customers move .

This is the gap between school and work .

It was 2 a.m. in the fourth week .

Lin Chuan sat in front of his computer, on the screen a database model he was building .

He drew a star-shaped model: in the middle is a sales fact table, surrounded by product dimension, time dimension, store dimension, and customer dimension .

He checked it twice to confirm there were no logical errors .

Then he started writing the table creation statements .

Write it, then execute it .

"Table created successfully."

He stared at the line of green text and suddenly realized something :

Three months ago, he couldn't even explain what SQL was .

Now, he is building a database .

This isn't theoretical . It's a real database that the team will use tomorrow .

He turned off the computer and lay down on the bed .

This time, he didn't suffer from insomnia .

He was too tired .

Studying in the early morning has one side effect: it makes his daytime blurry .

Once, he almost fell asleep during a station meeting. Mark asked him a question about data caliber, and it took him two seconds to react .

Another time, he wrote a query but forgot to add a WHERE clause, resulting in a full table scan. The database almost crashed. Daniel walked over, patted him on the shoulder, and said , " Didn't you sleep last night? "

Lin Chuan nodded .

Daniel said , " Don't try to learn too much too quickly. Data engineering is a marathon, not a sprint. "

Lin Chuan said , " I know. "

But he knew in his heart that he didn't know .

All he knew was that if he stopped now, he would be back in that winter of 127 rejection letters .

He didn't want to go back .

Chapter 8: The Invisible System

One day, Daniel came over and threw him a link .

" Take a look at this. "

Lin Chuan clicked on it. It was a technical blog post with the title :

Data Systems : What You See Is Only 10 %

There is a picture in the article :

· Above the water ( 10% ) : Reports, dashboards, analysis conclusions, AI models

· Below the surface ( 90% ) : Data acquisition, logging, message queues, ETL pipelines, data warehouses, metadata management, data quality monitoring, access control, disaster recovery .

He stared at that picture for a long time .

passage in the article :

" Most people 's understanding of data work remains ' above the surface . ' They think data analysis is just writing SQL , drawing charts, and running models. But what truly makes all of this possible is the 90% below the surface . The job of a data engineer is to build and maintain that 90% . No one will praise you for ' building a beautiful pipeline , ' but if the pipeline breaks down, everyone will come to you. "

Lin Chuan closed the link and walked to the window .

from the 12th- floor window , Toronto's streets resemble a circuit board. Traffic flows like data packets being transmitted, and crowds are like discrete nodes .

He suddenly had a strange feeling :

This city — this world — is full of invisible systems .

Electric power grid. Water pipes. Internet. Traffic signals. Banking system. Medical records .

Nobody thanks the power grid engineer when the lights are on. But when the power goes out, everyone's cursing .

The same applies to the world of data .

In the afternoon, Daniel took him to see something .

" Come, let me show you what ' below the surface ' really means . "

They entered the server room. It wasn't large, maybe twenty square meters, but it was cold — the air conditioning was set low and humming softly. Rows of black servers were stacked together, their blue indicator lights flashing .

“ This is our data warehouse server , ” Daniel pointed to one of them, “ all the analytical data is here. ”

Lin Chuan took a step closer. The server was black, with a label on it that read : " DW-PROD-01" .

" The SQL queries you write all the time end up running on this machine. "

Lin Chuan reached out and touched the server's casing. It was very cold .

He suddenly had a strange sense of connection — the code he wrote wasn't abstract logic, but something that was actually running on such a machine. This machine was consuming electricity, generating heat, and humming. It was located on the 12th floor of a glass building in downtown Toronto, in a server room with the air conditioning blasting cold, working day after day alongside dozens of other machines .

“ Let’s go , ” Daniel said, “ there’s one more place. ”

They left the server room, and Daniel led him to another office. There, several engineers were staring at screens displaying constantly scrolling logs .

" This is our data pipeline monitoring panel. "

Lin Chuan glanced at it. The screen displayed small squares of various colors — green for success, yellow for delay, and red for failure. It looked like a colorful river .

“ Look at this , ” Daniel pointed to a red square, “the Kafka consumer experienced a five-minute delay. This means the upstream data is being generated too quickly, and the downstream can't keep up. ”

" What is Kafka ? " Lin Chuan asked .

" Message queues. The first stop for data from the business system into this pipeline. You'll learn about them later. "

Lin Chuan nodded .

The first time he saw a " data system " was not an abstract concept, but a real thing with servers , monitoring panels, and red alarm lights .

That night, when he got home, he wrote a sentence in his notebook :

Analysts are " results -oriented people " .

Engineers are " results creators " .

The former sees only the iceberg above the water's surface. The latter builds everything beneath the surface .

He closed his notebook and turned on his computer .

Search : " What is Kafka ?"

Chapter Nine: The First Collapse

It was a Thursday afternoon .

Lin Chuan is currently working on a small task he took on — importing data from the CRM system into a data warehouse. It's not very complicated: every morning at dawn, he pulls data from the CRM API , cleans it, and loads it into the warehouse .

He wrote a Python script and ran it on a cron job . It ran for three days without any problems .

On the fourth day, something happened .

At 2 PM, his Slack suddenly exploded .

Daniel : @Lin Is the pipeline broken? The sales report has no data.

Mark : Lin Chuan, take a look, a customer has complained .

Emily : Lin Chuan, my analysis isn't producing the desired results.

He opened the monitoring panel .

A sea of red .

His script ran at 1 AM, but stopped after processing only two-thirds of the data. The error message was :

text

API rate limit exceeded. Retry after 3600 seconds.

The API is being rate-limited .

But the script has no retry mechanism. It stopped. And then nothing happened .

Data after 2 AM was not loaded .

This means that the reports the sales team saw this morning were based on yesterday's data. The customer success team saw customer information based on yesterday's status. The charts Mark presented at the management meeting were incorrect .

Lin Chuan's hands began to tremble .

It wasn't fear. It was something more complex — a clear sense that " I messed things up , " like an ice pick piercing my chest .

He sat down and began to fix it .

Step 1: Manually retrieve the missing data .

He wrote a Python script that fetched 1000 records at a time, then slept for 10 seconds to avoid triggering rate limiting again . It ran for 20 minutes and then finished fetching the data .

Step 2: Rerun the cleaning logic .

Step 3: Load into the data warehouse .

Step 4: Notify everyone that the data has been restored .

The whole process took an hour .

He sent a message to the group : " Data has been fixed, sorry. "

Daniel replied with a single word : " Okay. "

Mark did not reply .

Emily sent a private message : " It's okay, thank you for your hard work. "

That evening, he stayed in the office and didn't leave .

He sat at his workstation, staring at the error log .

text

API rate limit exceeded.

He asked himself: Why didn't the script retry ?

Because when he was writing the code, he didn't consider that the API would be rate-limited .

Why didn't he think of that ?

Because his mindset is still " academic mode "—in an academic environment, the data is given, the API is unlimited, the network is stable, and nothing will go wrong .

But the real world is not like that .

In the real world, APIs have flow limitations, networks can time out, databases can lock tables, disks can fill up, memory can be insufficient, timestamps can have time zone issues, character sets can have encoding problems, and data types can have implicit conversion issues .

There is no undo button in the real world .

You can't Ctrl+Z a pipe that 's already malfunctioning. You can only fix it. And while you're fixing it, business is suffering, customers are complaining, and the boss is checking his watch .

He wrote a sentence in his notebook :

writing code , always assume that something will go wrong .

It's not " if something goes wrong , " but " when something goes wrong . "

Then ask yourself: After an error occurs, will the system recover by itself? Or do I need to get up in the middle of the night ?

He added a line :

Next time I write a script, I'll add retry functionality, logging, alerts, and a dead-letter queue .

The next day, he reported this to Daniel .

After listening , Daniel didn't criticize him. He only said one sentence :

you know why we have a staging environment ?

Lin Chuan was stunned for a moment .

“ The goal is to identify issues before they impact production , ” Daniel said, “ but your script isn’t testing rate limiting in the staging area. Because you’re using the test API , there’s no rate limiting. ”

Lin Chuan understood .

His testing environment is different from the production environment. Something that works perfectly in testing might crash in production .

“ Next time , ” Daniel said, “ hardcode a simulated rate limiting switch in the code . Test it in staging and see what happens when the script crashes. ”

Lin Chuan nodded .

That day he learned a new term: chaos engineering .

It's not about causing damage, but about deliberately creating malfunctions to see if the system can survive .

That night, he couldn't sleep again .

It's not anxiety . It's reflection .

He replayed every detail of that afternoon in his mind over and over again :

· The script started running at 1 a.m.

· API rate limiting at 1:23 AM

· The script did not retry and exited directly.

· From 1:23 AM to 2 PM , no one knew there was a problem.

· 2 PM , the sales team discovered the report was incorrect.

· He manually repaired it between 2 and 3 pm .

· 3 p.m. , everything returned to normal.

He asked himself: If I could do it all over again, what could I improve on ?

· Retry mechanism

· Add an alert — if the script fails , send a message to Slack.

· Add a data quality check – if the daily morning report shows 20% less data than the previous day , an automatic alert will be triggered.

· Write a recovery script — no manual operation required, one-click repair.

He wrote all of this down in his notebook .

Then he fell asleep .

4:30 a.m.

Chapter 10 : The World of Engineers

When Lin Chuan arrived at the office on Monday morning , Daniel was already there .

“ Here , ” Daniel said, “ I ’ll buy you coffee. ”

They went downstairs to the coffee shop. Lin Chuan ordered an Americano, and Daniel ordered a latte . They sat by the window .

Daniel is in his forties , with some gray hair, but his eyes are bright. He has worked in the field of data engineering for fifteen years — in banking, e-commerce, and healthcare. He speaks slowly, but every word is crystal clear, without any redundancy .

“ Lin Chuan , ” he said , “ do you know why there’s such a shortage of data engineers? ”

Lin Chuan thought for a moment : " Because it's difficult? "

“ Because nobody wants to be one . ” Daniel chuckled. “ Data scientist sounds cool, machine learning engineer sounds cool, AI expert sounds cool. Data engineer? Sounds like a plumber. ”

Lin Chuan remained silent .

“ But do you know what the reality is ? ” Daniel took a sip of his coffee. “ Most companies don’t need machine learning at all . They don’t even have the basic data infrastructure. Giving them a deep learning model is less valuable than giving them a clean, reliable, and automatically updated data table. ”

" So what is the essence of a data engineer ? " Lin Chuan asked .

Daniel put down his cup and thought for three seconds .

" It's about structure. "

" Structure? "

" Yes. Data engineers aren't the ones who write code; they're the ones who design the structure. How you organize data, how you ensure data quality, how you make the data trustworthy—these aren't technical issues, they're structural issues. "

He continued :

“ Look at a city. Pipes, power lines, roads, networks — these are the structures. You can't see them, but they determine whether the city can function. Data systems are the same. A good data engineer builds structures that are invisible — because they are working, no one notices them. But a bad structure will have problems every day, and people will complain every day. ”

Lin Chuan recalled the afternoon last week when the pipes collapsed .

" So when I'm fixing the pipes , I'm actually fixing the structure ? " he said .

" Yes. But you're only fixing the surface. A real engineer fixes the structure itself, preventing the problem from happening again. "

Back in his office, Daniel showed Lin Chuan something .

It's a diagram. An architecture diagram of a data system .

Starting from the top left corner, the arrow points right, then down, then right again, like a subway map .

· Data sources : CRM , ERP , web logs, mobile SDK , third-party APIs

· Data Acquisition : Kafka , Flume , Sqoop

· Data storage : S3 , HDFS

· Data processing : Spark , Hive , Presto

· Data warehouses : Redshift , BigQuery , Snowflake

· Data modeling : dbt

· Data visualization : Tableau , Power BI , Looker

· Data quality : Great Expectations , dbt test

· Metadata management : Amundsen , DataHub

· Workflow Scheduling : Airflow

Looking at this picture, Lin Chuan felt as if he had entered a completely new world .

He only knew the small section on the far right before – Tableau , Power BI , and data visualization .

He didn't know there was such a long pipe on the left .

“ You don’t need to learn everything at once , ” Daniel said, “ but you need to know where they are and what they do. That way, when you encounter a problem, you’ll know which book to turn to. ”

That afternoon, Lin Chuan began a new task .

Daniel gave him a real need :

“ Our daily order data comes from two sources: one is pulled directly from Shopify , and the other is synchronized from the ERP system . The two sets of data often don't match. Your task is to write a data quality check script, run it daily, identify inconsistent orders between the two systems, and then generate a report. ”

Lin Chuan began dismantling the task .

1. Pulling order data from Shopify API

2. Retrieve order data from the ERP database

3. Comparison: Order amount, order status, customer information

4. Find the differences

5. Generate report

6. Send email

He spent two days writing the code. The first day he wrote the data acquisition and comparison logic. The second day he wrote the exception handling and report generation .

On the third day, he made another trip. He discovered 137 inconsistent orders .

Upon checking the details , he found that most of the discrepancies were due to time zone issues: Shopify uses UTC , while the ERP system uses EST . The order dates didn't match .

He added a time zone conversion. Then he ran it again. The inconsistencies dropped to 23 .

Those 23 were genuine issues: the order status was "shipped " in the ERP system , but " pending " in Shopify . He sent a report, copying the operations team .

Mark replied : " This is very useful. "

That evening, Lin Chuan returned home and sat down at the table .

He opened his notebook and wrote down a list of what he had been learning over the past few months :

SQL

· Window functions ✅

· Query optimization ✅

· Implementation Plan (Under Study )

Python

· Advanced pandas ✅

· API calls ✅

· Exception handling ✅

· Log (Under Learning )

Data Engineering Concepts

· ETL vs ELT ✅

· Dimensional modeling ✅

· Data quality ✅

· Kafka ( Just learned about it )

· Spark ( just learned about it )

· Airflow ( just found out )

He looked at the list and recalled three months ago .

Three months ago, he didn't even know SQL .

Three months ago, he didn't know what window functions were, what a data warehouse was, or what API rate limiting was .

Three months ago, he received his 127th rejection letter and felt that the world didn't need him .

Now, he sits at the same table, in the same chair, and outside the window, Toronto is still the same Toronto .

But things are different now .

It's not because he's acquired more skills .

Rather, it 's because—

He is no longer someone who " explains the world " .

He is becoming a " world builder " .

He closed his notebook and walked to the window .

In the distance, the lights of CN Tower are on, like a giant coordinate .

He recalled what Daniel had said earlier that day :

" Analysis is the last step, not the first. The first step is making the data analyzable. "

He used to think that data analysis started with " asking questions " .

Now he knows that before you can ask a question, you need a data system that can answer it .

who built that system were data engineers .

He turned off the light .

Lying in bed .

This time, he didn't suffer from insomnia .

He didn't think about " whether I will be replaced " .

He was thinking: Tomorrow, that data quality check script is going live. He needs to run it in the production environment .

He closed his eyes .

There was a slight curve at the corners of her mouth .

It's not a laugh .

It's the relief of finally finding direction .

Part Three: Restructuring ( Entering the Core Layer )

Chapter 11: Data Pipelines

After four months on the path of data engineering, Lin Chuan is about to embark on his first truly significant " major task " .

Daniel called him into the meeting room. It was just the two of them . The whiteboard had the words " Order Data Pipeline Reconstruction " written on it .

“ Lin Chuan , ” Daniel leaned back in his chair, twirling a pen in his hand, “ you’ve been here for almost four months. You ’ve almost mastered SQL , and you can write Python code now. I want to give you a complete project, from start to finish. ”

Lin Chuan sat up straight .

“ The current order data pipeline was built two years ago , ” Daniel said. “ Back then, we only had a few thousand orders a day, and it ran smoothly. Now we have 200,000 orders a day, and the pipeline crashes every day. I need you to redesign one. ”

Lin Chuan's heart raced. Not out of fear. It was because he had finally gotten this opportunity — not to repair someone else's pipeline, but to build his own .

" Are there any restrictions ? " Lin Chuan asked .

Daniel listed three points :

1. This cannot affect daytime business queries (meaning the pipeline must run in the early morning ).

2. Data latency cannot exceed four hours (meaning if the system crashes in the early morning, it must be restored before 9:00 AM ).

3. Data quality checks are required (dirty data must not be loaded ).

“ Also , ” Daniel stood up and patted him on the shoulder, “ I’ll be watching, but you’ll have to write the code yourself. ”

Lin Chuan spent two days designing .

He drew diagram after diagram in his notebook . The first version was too simple, lacking a retry mechanism. The second version was too complex, adding too many unnecessary features. In the third version, he finally drew an architecture that satisfied him :

text

Source Database -> Data Extraction -> Temporary Storage -> Data Cleaning -> Data Quality Check -> Data Warehouse - > Reports

↑ ↑ ↑ ↑

[ Retry Mechanism ] [ Error Log ] [ Exception Queue ] [ Monitoring and Alerts ]

He took the picture to Daniel . Daniel looked at it for two minutes and nodded .

" You can start writing now. "

For the next two weeks, Lin Chuan entered " immersion mode " .

He writes code at the company during the day . He continues writing at home in the evening. He even writes on weekends. He's like a machine; the input is coffee, and the output is Python code .

His first draft of the extraction logic was as follows :

Python

def extract_orders ( last_updated ):

query = f"""

SELECT * FROM orders

WHERE updated_at > ' { last_updated } '

"""

return run_query ( query )

But he soon encountered his first problem: if the order volume was large, this query would overwhelm the database .

He did some research and discovered a concept: incremental extraction . Instead of fetching the entire dataset each time, it only fetches the data updated since the last extraction. However, incremental extraction has a pitfall: if an order is deleted, you'll never know it's been deleted .

He added a soft deletion mechanism: instead of actually deleting the data, he added an is_deleted field .

The second issue: API rate limiting. He learned his lesson the hard way before . This time, he added a retry decorator to his code :

Python

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=4, max=60))

def call_api_with_retry(url):

response = requests . get ( url )

if response.status_code == 429 : # rate limit

raise Exception ( "Rate limited" )

return response

In this way, if the API is rate-limited, the script will automatically wait and then retry, up to five times .

The third problem: data cleaning. The source data contained various issues: null values, format errors, and duplicate data. He wrote a cleaning function :

Python

def clean_order ( row ):

#Handling null values

if pd . isna ( row [ 'customer_id' ]):

row [ 'customer_id' ] = 'UNKNOWN'

#Standardize date format

row [ 'order_date' ] = pd . to_datetime ( row [ 'order_date' ], errors = 'coerce' )

#Remove currency symbols from amounts

row [ 'amount' ] = row [ 'amount' ]. replace ( '$' , '' ). replace ( ',' , '' )

#Mark abnormal data

if row [ 'amount' ] <= 0 :

row [ 'is_anomaly' ] = True

return row

The fourth issue: data quality checks. He wrote down six rules :

1. 20% less than yesterday.

2. Order amounts cannot be negative.

3. Customer ID cannot be entirely empty .

4. Order dates cannot be after today.

5. No duplicate order numbers

6. from different channels should not fluctuate drastically.

If any rule fails, the pipeline will stop and send an alert to Slack .

Two weeks later, the code was finished. It had 847 lines of code .

Daniel had him run it in the staging environment for a week first .

On the first day, the pipeline crashed after running for two hours . The error message was: MemoryError . He investigated and discovered that he had loaded the entire dataset into memory while cleaning the data. Two hundred thousand rows of data, plus various intermediate variables, caused a memory overflow .

He changed to batch processing: process 10,000 lines at a time, write to the disk, clear the memory, and then process the next batch .

The pipeline ran successfully the next day. However, the data quality check returned an error: the number of orders was 25% lower than the previous day . He investigated and discovered that there had been a one-hour write delay in the source database from the previous day , causing some orders to miss the extraction. He adjusted the extraction time window, adding a half-hour buffer .

On the third day, the pipeline was running smoothly. Data quality checks also passed. However, the reporting team said there was a problem with the data they were seeing — the amounts for some orders were incorrect. He investigated for three hours and discovered a timezone conversion bug : the order times were UTC , but his data cleaning function was using local time. He fixed it .

On the fourth day, the pipeline finished running. The data was correct. But it was too slow — it took four hours . He added some indexes, optimized the JOIN order, and changed some Python logic to SQL . It ran for two hours .

Day 5, two hours . Day 6, two hours. Day 7, two hours .

It's stable now .

On Monday morning, Daniel walked to his workstation .

"Staging has been running for a week, the data is correct, and the performance is stable. We'll go live to production next Monday. "

Lin Chuan nodded . But his heart was beating faster than usual .

“ On the day it goes live , ” Daniel said, “ I’ll be there . But you’ll be operating it. ”

Monday. 4:00 AM .

Lin Chuan arrived at the office, it was still dark. The streetlights on Yonge Street cast an orange glow on the snow. He swiped his access card to open the door and walked into the empty office .

Daniel was already there. He was sitting at his workstation with a cup of coffee in front of him .

" Ready ? " Daniel asked .

Lin Chuan took a deep breath . " Ready . "

He opened the terminal and connected to the production server .

Step 1: Back up existing data. He ran a backup script that copied the current production data to a backup table. Thirty seconds later, it was complete .

Step two: Deploy the new code . He used git to merge the code from the staging branch to the production branch. Ten seconds .

Step 3: Run a test run. He set a flag to make the pipeline only process data from the last ten minutes and not write it to the production table. Run. Thirty seconds. Done. Data correct .

Step 4: Official Run. He turned off the " Test Run " icon and set the processing window to " Past 24 Hours " . Then he pressed Enter .

The log begins to scroll on the screen :

text

[INFO] 2024-03-15 04:15:23 - Starting data pipeline

[INFO] 2024-03-15 04:15:24 - Extracting orders from source

[INFO] 2024-03-15 04:15:45 - Extracted 23,847 orders

[INFO] 2024-03-15 04:15:46 - Starting data cleaning

[INFO] 2024-03-15 04:16:12 - Data cleaning completed

[INFO] 2024-03-15 04:16:13 - Running data quality checks

[INFO] 2024-03-15 04:16:15 - Quality check passed

[INFO] 2024-03-15 04:16:16 - Loading to data warehouse

[INFO] 2024-03-15 04:16:52 - Load completed

[INFO] 2024-03-15 04:16:53 - Pipeline finished successfully

Lin Chuan stared at the last line, his hands still on the keyboard .

" Did it work ? " he asked .

Daniel glanced at the screen and nodded . " It worked. "

Lin Chuan leaned back in his chair. His shoulders suddenly relaxed — he had been tense all over without realizing it .

“ But the real test , ” Daniel said, “ is today during the day. See if the reporting team has any complaints. ”

People started arriving at the office at nine o'clock in the morning .

Emily was the first to arrive. She opened Tableau and refreshed the reports .

“ Wow , ” she said , “ the data came out so early today. Before, we had to wait until ten, but now it’s available at nine. ”

Lin Chuan didn't say that it was his pipe. But the corner of his mouth twitched involuntarily .

At 10:00, Mark will hold a meeting .

“ The sales report is on time today , ” Mark said. “ Who made it? ”

Daniel glanced at Lin Chuan. Lin Chuan hesitated for a second, then said , " I restructured the order pipeline. "

Mark nodded . " Good . "

Just two words .

But Lin Chuan felt that these two words carried more weight than any lengthy praise .

That night, he returned home .

He opened his notebook and wrote the first line under " Completed Projects " :

Order data pipeline reconstruction ✅

He stared at that line for a long time .

This is not an assignment. It is not an exercise. It is not a paper .

data pipeline that runs in a production environment, processes 200,000 orders every day, and allows the sales team to see reports on time .

For the first time, he felt that he was " changing the system " rather than " explaining the system " .

He wasn't analyzing the data .

He is building something that makes data analytics possible .

Chapter Twelve: Behind AI

After the order pipeline had been running stably for a month, the company decided to do something " big " : introduce an AI prediction model .

The goal is to predict sales volume for the next seven days and help the supply chain team prepare inventory in advance .

Mark announced the news at the all-hands meeting, his tone was clearly excited :

“ We will use machine learning to optimize our supply chain. This will reduce our inventory costs by 15% and our stockout rate by 30% . ”

Everyone applauded .

Lin Chuan clapped as well. But a question lingered in his mind: Was the data ready ?

The AI project leader is a data scientist named Sophia . She has only been with the company for two weeks, and previously worked on recommendation systems at an e-commerce company. She speaks quickly, likes to use PowerPoint , and each slide contains more than three terms like " deep learning, " " neural network ," and " feature engineering . "

At the first project meeting, Sophia stood in front of the whiteboard and drew a flowchart :

text

Historical sales data -> Feature engineering -> Model training -> Predicting future sales -> Inventory optimization

“ We need data , ” Sophia said, “ at least three years of historical sales data, broken down by day, plus promotional information, holiday data, and weather data. ”

Mark nodded . " Lin Chuan, can you provide this data? "

Lin Chuan said , " We have sales data. But the promotional information is incomplete, and the weather data needs to be pulled from an external API . "

“ Okay, let’s do it , ” Mark said. “ How long will it take to get ready ? ”

Lin Chuan thought for a moment : " Two weeks. "

" I'll give you a week. "

Lin Chuan began preparing the data. This was the first time he had prepared training data for an AI model .

He quickly discovered a problem: there was a lot of " noise " in the sales data .

· Last year on Black Friday, the system had a bug , and order data was lost for two hours.

· In January of this year, a warehouse was relocated, and there were no shipments for three days.

· One day in March, the website was down for two hours , and the number of orders plummeted.

· There were also some promotional activities where the data wasn't recorded; all we know is " there was a discount that week ."

If you feed this data directly to the model, the model will learn the wrong things — it will think that " there is one day in March every year when the order volume is particularly low , " but that is caused by the website going down and has nothing to do with sales patterns .

He went to find Sophia .

" How should we handle this abnormal data? "

Sophia glanced at it and said , " Mark them. When we build the model, we can add a weight to reduce the impact of these points. "

Lin Chuan asked , " Are you sure the model can handle it correctly ? "

Sophia smiled . " Don't worry , the XGBoost we're using is very robust . "

Lin Chuan didn't refute it . But a voice in his mind said: No matter how robust the model is, it can't save flawed data .

In the second week, he started pulling weather data .

He used a free weather API and wrote a script to pull historical weather data for Toronto every day — temperature, precipitation, humidity, and wind speed .

But he soon discovered a second problem: weather data is measured in " days , " while sales data requires " order time . " If a customer places an order at noon , should the weather data be from 11 AM or 1 PM ?

He didn't know .

He went to ask the sales representative. The sales representative said , " We 're not sure either. Maybe ... it's the weather that day? "

The word " probably " made Lin Chuan's nerves tense up .

After thinking about it, he made a decision: to keep both the " weather on the day the order is placed " and the " weather the day before the order is placed " in the data table , and let the model learn which one is more important .

This isn't a perfect solution. But given the incomplete information, it's the best option he can take .

By the third week, the data was finally ready .

Sophia began training the model. She used a very complex feature engineering approach — breaking down the time features into year, month, week, day, day of the week, whether it was a weekend, whether it was a holiday, and also adding lagged features (sales figures from the previous day, the previous week, and the previous month) .

The model ran all night. The next day, Sophia walked into the office excitedly :

" Validation set accuracy: 92% ! "

Everyone crowded around to look. On the screen was a beautiful chart — the predicted and actual values almost perfectly matched .

Mark was also very happy . " When will this model be available? "

Sophia said , " If the data is fine, it can be launched in two weeks. "

Lin Chuan stood at the back of the crowd without saying a word .

He noticed a detail on the chart: the model's predictions were particularly accurate around holidays. But he knew that half of the promotional activities in those holiday sales data were not recorded .

The model doesn't " learn " the rules . The model " remembers " history .

He contacted Daniel privately .

you think that 92 % is believable ?

Daniel leaned back in his chair and thought for a while .

" A 92% accuracy rate on the validation set doesn't mean it will be 92% in the production environment . Do you know why? "

Lin Chuan shook his head .

" Because the validation set and training set come from the same distribution. But the data distribution in a production environment changes. Holidays change, promotions change, competitors change, user behavior changes. The model learns from the past, not the future. "

" What should we do then ? "

" You need to help Sophia perform a ' data distribution drift detection' — check daily whether the data distribution in the production environment matches the training data . If the drift is too large, the model's predictions will be unreliable. "

Lin Chuan nodded .

He went back and wrote a data quality monitoring script to compare data daily .

· daily order volume vs. distribution of training data

· Daily percentage of each channel vs. percentage of training data

· Average order amount per day vs. average value of training data

If the difference exceeds the threshold, an alarm will be triggered .

Two weeks later, the model went live .

in the first week . The supply chain team adjusted inventory based on the model's recommendations .

In the second week, one product category suddenly sold exceptionally well, something the model hadn't predicted. The reason was that a competitor ran out of stock in that category, and customers flowed to them .

The model doesn't " know " that competitors are out of stock because it doesn't have that data .

Lin Chuan looked at the alarm on the monitoring panel: the order volume distribution was drifting, with a deviation of 47% .

He messaged Sophia : " The data has drifted; the model may need to be retrained . "

Sophia replied : " Okay, I'll take a look. "

That night, Lin Chuan wrote a sentence in his notebook :

AI is not magic. AI is a function of data .

If the data is wrong, incomplete, or outdated, then the AI is wrong, incomplete, and outdated .

a data engineer is to ensure that AI stands on a reliable foundation .

He added another sentence :

Beneath the surface lies the true engineering feat .

Chapter Thirteen: The Illusion of Decision-Making

AI model went live, something happened. This event gave Lin Chuan a completely different understanding of the phrase " data -driven decision-making . "

It was a Wednesday afternoon. Mark called an emergency meeting .

“ The model shows that user satisfaction has increased by 8% in the past two weeks , ” Mark said, standing in front of the whiteboard with a smile. “ This shows that our recent redesign has been effective. Supply chain inventory preparation is more accurate, and user wait times have decreased. ”

Sparse applause rang out in the conference room .

Sophia added , " The model predicts that satisfaction levels will continue to rise over the next month, by approximately 5% to 10% . "

Mark said , " Okay, then we 'll increase our investment. Next quarter, we'll expand AI predictions to all product categories. "

Lin Chuan sat in the corner, remaining silent .

It wasn't because he was unhappy. It was because he had just been doing a data quality check yesterday and discovered a problem .

He hesitated for a moment. Should he speak? Speaking would disrupt the meeting. But what if he was wrong and didn't speak ?

He raised his hand .

“ Lin Chuan ? ” Mark looked at him .

" I ran a data quality check yesterday and found a change in the source of the user satisfaction data. "

" What changes? "

" Two weeks ago, the IT team changed the trigger logic for the user satisfaction survey. Previously, the survey was sent to 10% of users randomly after they completed an order . Now, it's sent to all users after all orders are completed. "

The meeting room fell silent .

“ This means , ” Lin Chuan continued , “ that the current satisfaction data is not based on the same sample as the data from a month ago. A direct comparison could be misleading. ”

Mark frowned . " You mean, satisfaction levels might not have increased? "

" I don't know if there's been an increase. I'm just saying this is rather unfair. The two datasets used different sampling methods. "

Sophia frowned as well . " The sampling method has changed, but the model has been tuned using feature engineering ..."

“ Feature engineering cannot solve sample bias , ” Lin Chuan said . He knows he’s now challenging a data scientist, but he can’t stop. “ If the representativeness of the sample changes, any model will be misleading. ”

There was a five-second silence in the meeting room .

Mark said , " Lin Chuan, do you have data to prove that satisfaction levels haven't increased? "

" No, I didn't. I was just raising a data quality issue. "

" Then let's proceed according to the original plan. Sophia , please double-check the data definitions later . "

Lin Chuan sat in the chair, his palms sweating .

He knew he was right. But he also knew that in this meeting room , being " right " wasn't necessarily a good thing .

After the meeting ended, Daniel came over .

“ You ’re right , ” he said , “ but there could be a better way. ”

" What's the meaning? "

" By saying ' this data might be problematic ' in the middle of the meeting , you've essentially negated the entire AI project's achievements. Sophia and Mark both invested a lot in that project. You should have spoken to them privately beforehand. "

Lin Chuan opened his mouth, wanting to argue. But he knew Daniel was right .

“ I’ll be more careful next time , ” he said .

Daniel nodded . " However, it's commendable that you had the courage to say it in the meeting. Most people wouldn't do that . "

A week later, Sophia reanalyzed the data .

She found that if only the satisfaction levels of " users sampled in the same way " were compared , the actual change was +1.2% , not +8% . Moreover, this change was not statistically significant .

She posted a message in the Slack group :

" The adjustment in data caliber affected the results; the previous 8% was an overestimation. The actual change was about 1% , which is not significant. We recommend temporarily suspending increased investment. "

Mark replied with a single word : " Okay. "

There was no blame. No apology. Just a simple " okay " .

But Lin Chuan knew that if he hadn't said that at the meeting, the company might have actually invested hundreds of thousands in a " false growth . "

That night, he wrote a long passage in his notebook :

Data doesn't lie, but people do .

It wasn't a malicious lie. It was an unintentional misinterpretation .

The sampling changed, but nobody noticed. The criteria changed, but nobody updated the documentation. The model produced beautiful results, but nobody asked, " Where did the data come from ? "

Data- driven decision-making sounds very scientific. But data itself is defined, collected, and interpreted by people .

every metric lies a choice: What caliber to use? How to sample? How to clean the samples? How to normalize them ?

These choices determined the conclusion .

a data engineer is not to “ provide the correct data ”—because there is no such thing as absolutely correct data .

a data engineer is to make these choices transparent. To let decision-makers know: where the data came from, what its limitations are, and what questions it cannot answer .

He added another sentence :

When a number is placed on a screen, it acquires the authority of " truth, " even if it is wrong .

Don't let data become a new superstition .

Chapter Fourteen: The Optimized Person

That autumn, the company started a new project: efficiency optimization .

The name sounds nice. But everyone knows what it means — layoffs .

The news leaked from a management meeting. Someone in the break room said that Mark was meeting with HR to discuss " organizational restructuring . " Someone else saw a message on Slack that shouldn't have been seen: " We need to reduce our personnel costs by 15% . "

Lin Chuan first heard the news, he was writing a new data pipeline. He paused for a moment, then continued typing .

It's not because he doesn't care. It's because he doesn't know what to do .

efficiency optimization is not layoffs, but " automation " .

The company introduced a new BI tool that can automatically generate 90% of routine reports. Reports that used to require an analyst's entire day to create can now be done with just a few clicks of the mouse .

Lin Chuan looked at the tool and saw it generate a report that he had spent two months maintaining, and he felt a complicated feeling .

He recalled his fear when he first started working : " If machines can do my job, what will I have left? "

Now, that fear is becoming a reality — not for him, but for the team of analysts .

Emily 's first report was automated. Then the second. Then the third .

One day at noon, Lin Chuan was getting water at the tea station when Emily came over .

“ You know , ” she said , “ our team’s reporting automation rate has reached 80% . ”

Lin Chuan nodded .

"Mark said we don't need to hire any more new people. Existing analysts need to transition to ' business insights , ' and can no longer be just data-collecting tools. "

" Do you think you can turn it ? " Lin Chuan asked .

Emily smiled, but there was something in that smile he had never seen before — not sadness, not anger, but an inexplicable weariness .

“ I tried learning SQL and Python , but progress was very slow. I have too many meetings during the day, and I have to take care of the kids when I get home at night. My husband is away on a business trip in Calgary, and I'm taking care of two kids by myself …”

She didn't finish speaking. Lin Chuan didn't know what to say either .

The second wave was outsourcing .

The company outsourced part of its data cleaning work to an Indian IT services company. The cost was only one-third of that of local engineers .

Lin Chuan's team was unaffected — because the data pipeline requires 24/7 on -call service , which the outsourcing company couldn't provide. However , two junior analysts on the analyst team were notified that their contracts were expiring and would not be renewed .

That afternoon, Lin Chuan saw the two analysts packing up their things. One was a recent Indian woman who had only been working for eight months. The other was a local man who had been with the company for two years; his workstation was covered with sticky notes .

They didn't say anything. They just quietly put the things into the cardboard box .

Lin Chuan wanted to say something, but he didn't know what to say . " I'm sorry " ? It wasn't his fault. " Good luck " ? That was too lenient . " Keep it up " ? That was too insincere .

He didn't say anything . He went back to his workstation and continued writing code .

The third wave will be the real layoffs .

On the first Monday of November, everyone received an email: General Assembly at 10:00 AM today .

The conference room was full. Mark stood at the front, his face paler than usual .

“ The company is undergoing a strategic adjustment , ” he said . “ We need to make the organization leaner and more efficient. This means that some positions will be eliminated. ”

He read out a list .

Eleven names .

Lin Chuan heard a familiar name .

Emily .

After the meeting , Lin Chuan went to find Emily .

She was at her workstation, putting things into cardboard boxes. The small cactus. The photo of the dog. Several books on data analysis. A water cup with the company logo on it .

“Emily…”

She looked up at him. Her eyes were red, but she wasn't crying .

“ It’s alright , ” she said . “ I already knew. ”

you know ?

“ I saw my name on Mark ’s calendar . He invited me to a meeting with HR . If it were about a promotion, he wouldn’t have called HR . ”

Lin Chuan remained silent for a few seconds .

are your plans next ?

Emily put the last book into the box and stood up .

" I'll settle the kids down first . Then I'll look for a job. I might go to a small company, doing operations or sales support. Data analysis might not be a good fit for me. "

She picked up the box and walked outside .

she reached the door, she stopped and looked back at Lin Chuan .

“ Lin Chuan , ” she said , “ you’re different . You’re in engineering. They can’t touch you. ”

" Why? "

" Because you're the one building the pipeline. They 're laying off the people who use the data, not the people who create it. "

She smiled .

Then I got into the elevator .

Lin Chuan stood by the window , watching her walk out of the building. She carried the cardboard box, walked a short distance on the sidewalk, stopped, put the box on the ground, switched hands, and continued walking .

Winter has come to Toronto again. The wind is strong. Her hair is all messed up .

She didn't turn around .

Lin Chuan stood there for a long time .

Then he returned to his workstation, opened his notebook , and wrote down a sentence :

It's not ability that determines whether someone stays or leaves, but rather their " irreplaceability " .

If what you do can be automated , outsourced, or standardized , then you are replaceable .

It's not because you didn't try hard enough. It's because this system doesn't reward effort .

This system rewards you for things that only you can do .

He closed his notebook .

Start writing code .

It wasn't because he wasn't sad. It was because he was afraid — afraid that one day, his name would appear on such a list .

Chapter Fifteen: Layoff List

The office was much quieter during the first week after the layoffs .

The empty workstations haven't been cleared. The computers are still there, the chairs are still there, but the people are gone. It's like a hard drive with data deleted but space not yet reclaimed .

Every time Lin Chuan passed Emily 's workstation, he would see that little cactus still there. She had forgotten to take it with her. He hesitated a few times about whether to throw it away, but in the end, he left it there .

He didn't know why .

Perhaps it's because that cactus is the only thing still alive in this workstation .

At the first team meeting after the layoffs , Mark said the following :

" I know things are tough lately . But we have to look forward. Those who remain are the core team. The company needs you. "

Lin Chuan noticed that Mark's eyes flickered when he said " core team " .

After the meeting, Lin Chuan went to Daniel 's workstation .

"Daniel , I have a question for you. "

" explain. "

Do you think I'll be laid off ?

Daniel put down the mouse and turned to look at him .

" You want to hear the truth ? "

" think. "

" You won't be laid off now because the data pipeline you're building is a core system , and no one else will maintain it. But if you don't do something, you might be laid off in three years. "

" What's up? "

Daniel said , " Make yourself irreplaceable, rather than being trapped in an irreplaceable position. "

Lin Chuan frowned . " Is there a difference between the two? "

" Yes. The former is your ability that makes you valuable anywhere. The latter is that the system can't function without you, but without the system you are nothing. "

Lin Chuan remained silent for a long time .

Daniel continued , “ You ’re the one who ‘ fixes this pipe ’ now . But what if one day this pipe doesn’t need fixing anymore? What if the company switches to a new system? What if the company goes bankrupt? Will you still be irreplaceable? ”

Lin Chuan shook his head .

“ So , ” Daniel said, “ you’re not just learning how to fix this pipe. You’re learning — how to design any pipe. How to understand any data system. How to solve any data problem. Not ‘ irreplaceable in this company , ’ but ‘ irreplaceable in this industry . ’ ”

Lin Chuan nodded .

That night, he returned home but did not open the code editor .

He opened a job search website. Not to look for jobs, but to look at job descriptions .

He looked at the job requirements for ten data engineers . He listed them together and looked for commonalities :

· SQL ( 100% )

· Python ( 90% )

· Data warehouse ( 80% )

· Cloud platform ( 70% )

· ETL/ELT ( 90% )

· Data modeling ( 80% )

· Airflow/dbt ( 60% )

· Spark ( 50% )

He realized he had already mastered the first four. He wasn't very familiar with the last four yet .

He opened the learning website and wrote the following in the " Next Learning Goal " section :

Airflow → dbt → Spark

He wrote another sentence in his notebook :

Those who are laid off are not necessarily the least capable people, but rather the people who are most easily replaced .

Being easily replaceable doesn't necessarily mean you lack ability. It could be because your work is too standardized. It could be because your skills are too generic. It could be because you haven't built a " only you can solve " competitive advantage .

A moat isn't about hiding things. It's not about locking documents in a drawer. It's not about writing code that only you can understand .

The real moat is: your ability to solve problems outpaces the rate at which problems arise .

He closed his laptop and turned off the light .

Toronto , as seen from the window , is entering its coldest season of the year .

snowed yet. But it will soon .

He knew Emily wouldn't come back. Neither of the two analysts would come back. And perhaps one day, he wouldn't come back either .

But the difference is—

He is no longer someone who is " selected by the system " .

He began to become a person of " selection system " .

Part Four: Breakthrough (Individual and Era )

Chapter Sixteen: Leaving the System

Three months after the layoffs , Lin Chuan submitted his resignation .

It wasn't because he was laid off. It was because he had come to a realization .

It was March 15th, and the snow in Toronto was finally starting to melt. The snowdrifts along the sidewalks had turned into small gray hills, and water was flowing down the drains with a rushing sound. There was a different smell in the air — not the chill of winter, but the earthy scent of spring .

He walked into Mark 's office and placed the printed resignation letter on the desk .

Mark glanced at it and looked up .

" Why? "

Lin Chuan sat down. He thought for a long time about how to answer the question. Finally, he told the truth .

" I want to do something of my own. "

Mark paused for a few seconds . " Your pipeline is running very well. I was planning to promote you next quarter. "

“ Thank you , ” Lin Chuan said , “ but I need to know if I can survive without relying on a company. ”

Mark looked at him with something indescribable in his eyes — perhaps understanding, perhaps regret, perhaps a middle-aged man's envy of a young man .

“ If you ever want to come back , ” Mark said, “ the door’s open. ”

" Thanks. "

Lin Chuan was tidying up his workstation , Daniel walked over .

" Have you decided? "

" It's decided. "

Daniel extended his hand. Lin Chuan grasped it .

“ You’re the best junior I ’ve ever mentored , ” Daniel said, “ not because of your skills, but because you always ask ‘ why ’ . ”

Lin Chuan smiled . " You taught me. "

Daniel shook his head . " I didn't teach you. You already knew it. I just didn't extinguish it . "

They stood by the window, looking down at Yonge Street. The traffic and pedestrian flow were as usual, and no one noticed a data engineer leaving a glass building .

" What's next ? " Daniel asked .

" Take on some freelance projects. Try freelancing. "

" And the first customer ? "

Lin Chuan pulled a business card from his pocket . The card was simple: white cardstock with black lettering .

Lin Chuan / Data Consulting

Below is a phone number and an email address .

“ I only have one , ” Lin Chuan said . “ Here you go. ”

Daniel glanced at it, smiled, and put the business card in his pocket .

“ The first project , ” he said , “ I’ll help you find one. ”

A week later, Lin Chuan sat in a coffee shop , across from a man named Mike .

Mike , in his early forties , is the founder of a small e-commerce company. He sells outdoor gear — tents, sleeping bags, and trekking poles. The company isn't large , with annual sales of about two million Canadian dollars , but growth has recently stalled .

“ Our problem is simple , ” Mike said, “ customers buy once and never come back. The repurchase rate is only 15% . I don’t know why. ”

Lin Chuan opened his laptop . " Do you have the data ? "

" Yes. Shopify has all the order data in its backend. And Google Analytics too . But I don't know how to analyze it. "

“ Give me a week , ” Lin Chuan said , “ I’ll take a look. ”

" How much? "

Lin Chuan hesitated for a moment. This was his first offer. He didn't know how much to say .

“ One thousand , ” he said .

Mike thought for a moment . " Okay. If we can find the problem, it's worth the money. "

Lin Chuan returned home and began his first independent project .

He spent a day exporting the data: Shopify order data, customer data, and product data; Google Analytics traffic data and behavioral data .

Then he spent two days cleaning the data. Just like before at the company — null values, duplicates, formatting issues. He was very familiar with this process. Only this time, he didn't have to report to anyone. He was only responsible for the results .

On the third day, he began to analyze .

He asked himself a question: Why aren't the customers coming back ?

He first reviewed the customers ' initial purchasing behavior. He divided the customers into two groups: those who returned and those who didn't .

The comparison revealed :

· Returning customers' average first purchase amount was $87

· For customers who didn't return , the average first purchase amount was $42.

More than double .

He then looked at the product categories. Returning customers bought mostly " tents " and " sleeping bags . " Customers who didn't return bought mostly " trekking poles " and " water bottles . "

He called Mike .

“ I ’ve discovered a pattern , ” Lin Chuan said . “ Customers who buy cheap items don’t come back. Customers who buy expensive items do. ”

" Why ? " Mike asked .

" I guess it's because people who buy trekking poles might only go hiking occasionally. People who buy tents are true outdoor enthusiasts. The former don't have a need for repeat purchases, while the latter do. "

There was silence for a few seconds on the other end of the phone .

" So you mean my clients aren't the same type of people? "

" Yes. You have two completely different customer groups. You need to treat them differently. "

On the fourth day, Lin Chuan gave his advice :

1. For " occasional users " : Don't expect them to make repeat purchases. However, when they make their first purchase, you can recommend related products with a higher average order value (for example, recommend a tent to someone who buys trekking poles) .

2. For " core users " : Establish a membership system and offer discounts on repeat purchases. Send emails to push new product information and outdoor guides. Build a community of outdoor enthusiasts on social media .

3. Adjust your ad placement : Shift your budget from general outdoor keywords to more precise keywords like " camping, " " tent, " and " hiking . "

Mike listened and remained silent for a long time .

How do I know these suggestions are effective ?

“ Let’s do A/B testing , ” Lin Chuan said , “ Let’s change one channel and see how the data changes. ”

" Okay. I 'll give it a try. "

Two weeks later , Mike called .

" Conversion rate improved by 8% . Advertising ROI went from 2.5 to 3.2 . Thank you. "

The next day, Lin Chuan received a transfer of $2,000 into his bank account .

$1,000 more than they had agreed upon .

Lin Chuan stared at that number for a long time .

$2,000 .

This is not salary. It is not a bonus. It is not " compensation paid by the company to its employees " .

This is the money he earned by using his abilities to directly solve someone's problem .

No middlemen . No HR . No performance reviews. No " we have a limited budget . "

Only problems, solutions, and results .

That night, he wrote a sentence in his notebook :

The value of data lies not in its analysis, but in " changing the outcome " .

At the company, he created reports, built pipelines, and performed data quality checks. These tasks were valuable, but indirect. He had no idea how much extra money his code ultimately helped the company earn .

But here he saw: a suggestion that increased the conversion rate by 8% . 8% was money. The money was real. Real enough that he could feel it .

He added another sentence :

In large companies, you are a function. The input is the task, and the output is code. You don't know where your output goes or what impact it has .

In a small company, you are the entire process. From problem to solution, from data to decision, from code to revenue .

The latter is more tiring, but also more authentic .

Chapter Seventeen: The First Income

With his first income, Lin Chuan began to seriously pursue freelancing .

He built a simple website. The homepage only had three lines of text :

Lin Chuan

Data Engineer / Analyst

problems with data

He put a contact form at the bottom and then started posting on social media .

No one contacted him during the first week .

The following week, someone contacted me. It was a girl named Sarah who owned a yoga studio . Her question was: Why is my Instagram ad conversion rate getting lower and lower ?

Lin Chuan helped her analyze three months of data and discovered a problem: her ad creatives hadn't been updated. Users were tired of seeing the same video after three months .

He suggested she change her content every two weeks and provided three different creative directions. Sarah adopted his suggestions . A month later, the conversion rate rebounded by 25% .

Sarah paid him $800 .

In the third week, another customer came in . He was the owner of a pet supply store. His problem was: my inventory is always either too much or too little .

Lin Chuan helped him build a simple sales forecasting model. Using sales data from the past two years, it predicted the demand for the next month. Although the model was very simple — just a moving average with seasonal adjustments — it was much more accurate than the boss's " gut feeling " when it came to stocking up .

Inventory turnover increased by 18% . The customer paid him $1,500 .

A month later, Lin Chuan calculated his income :

· Outdoor gear store: $2,000

· Yoga Studio: $800

· Pet supply store: $1,500

Total : $4,300

than his previous monthly after-tax salary at the company .

And he only worked for fifteen days. The remaining fifteen days were spent studying , sleeping, and strolling through Toronto's parks .

For the first time, he felt that time was his own .

But soon, he encountered a problem .

The number of customers is increasing, and their needs are becoming more and more diverse. He can't handle it all by himself .

One day, he was doing three things at the same time :

· Write SQL for a yoga studio

· for pet supply stores

· Helping a new client — a small restaurant — analyze takeout order data

His Slack rang from 8 a.m. to 10 p.m. His inbox was overflowing with unread emails. His notebook was covered with sticky notes, each labeled " To Do " .

He was back to being " chased by missions " .

however , it wasn't his boss who was pursuing him, but a client .

The difference is: before, he couldn't refuse. Now he can .

But the problem is — he doesn't want to refuse . Every customer means money. Every refusal means losing money .

He went to find Daniel for coffee .

“ You’ve come to a problem , ” Daniel said. “ A very good problem. ”

" What's the problem? "

" You don't have enough time. That means your service is valuable. If no one is contacting you, that's the problem. "

" Then what should I do? "

Daniel took a sip of his coffee . " Two choices. First, raise prices. Second, productize your service. "

" Productization? "

" What you 're selling now is time. One hundred dollars an hour. But your limit is 24 hours a day. If you want to earn more, you can only sell it for a higher price. But if you can sell a ' thing' — a software, a tool, an automated process — you can replicate it infinitely. Sell it once, earn money once. Sell it a hundred times, earn money a hundred times. "

Lin Chuan thought for a moment . " You mean , turn what I'm doing into a product? "

" Yes. You help clients analyze data and give suggestions. How much of that advice is repetitive ? "

Lin Chuan thought for a moment .

at yoga studios , pet supply stores, and restaurants — though the industries differ, the essence is the same: customers want to know " what happened, why, and what to do . "

He can create a tool that automates 80% of the work .

That night, he opened his code editor and started writing a new project .

This is not a script for a specific client .

It is a product .

He named it: DataPulse .

The function is very simple :

1. Connect to the client's database or API

2. Automatically generate core indicator dashboards

3. Automatically detect anomalies (sudden drop in sales, sudden increase in traffic ).

4. Automatically generate a " Data Insights of the Week " report, outlining the three most important findings in natural language.

He spent three weeks writing the first version. It was rough. It wasn't fully functional. But it worked .

He sent emails to three clients he had previously served :

" I created a tool that can automatically generate data reports. Free trial for one month. After that, it 's $99 per month . "

three clients replied .

" Okay, I 'll give it a try. "

Chapter 18: The Birth of the Product

DataPulse was used by three clients .

Lin Chuan checks the backend usage data every day. Who logged in? Who used which function? Who didn't return ?

He discovered some patterns :

· Sarah from the yoga studio would check the dashboard once a week , and then she wouldn't come back .

· The pet supply store owner logs in every day, but only uses the " Sales Overview " page .

· Mike from the outdoor gear store only used it twice and then stopped .

He called Mike .

"Mike , why aren't you using it anymore? "

“ The tools are great , ” Mike said, “ but I don’t know how to use the data. You show me ‘ conversion rate dropped by 5% ,’ but I don’t know what to do. What I need is advice, not numbers. ”

Lin Chuan hung up the phone and sat there thinking for a long time .

Mike is right .

Data itself has no value . Data only has value when combined with decision-making .

His tool only did the first half — displaying the data—but not the second half — providing suggestions .

He started modifying the product .

The second version added a new feature: intelligent suggestions .

The rules are simple :

· If sales drop by more than 10%, it is recommended to " check if any promotional activities have ended or competitors have launched new products in the past week ".

· If the repurchase rate is below 20%, it is recommended to " consider launching membership discounts or email repurchase campaigns ".

· If a product's sales suddenly drop , it is recommended to " check if the product is out of stock or has been removed from the shelves ".

It's not AI . It's just an if-else rule. But it's sufficient for small business owners .

He then changed the price from $99 to $199 .

The addition of the suggestion feature has increased its value .

The following month, all three customers renewed their subscriptions .

The owner of the pet supply store also introduced him to a new customer — his friend who owned a bicycle shop .

By the fourth month, DataPulse had twelve paying customers .

Monthly income: $2,388 .

Including his ongoing custom consulting work, his total monthly income has reached around $8,000 .

Lin Chuan calculated that this was twice the amount he earned at work .

But his joy didn't last long .

Because he discovered a new problem .

DataPulse 's customers canceled their subscriptions after one month .

He checked the data and found that the customers who cancelled were those who " logged in infrequently and stopped using the service after two weeks " .

Why aren't you using it anymore ?

He called a customer who had cancelled. The customer said :

" The tools are great. But I'm too busy to check the data weekly. And, to be honest, your suggestions aren't always accurate. You suggested I try email marketing, which I did, but it didn't work. "

Lin Chuan hung up the phone .

He knew where the problem lay .

His " intelligent suggestions " are based on general rules. However, every industry and every client is different. General rules may be right or wrong in specific scenarios .

What he needs is not rules .

What he needs is to truly understand each client 's business .

But this brings us back to the old problem: if he takes the time to understand each client's business, he's selling time, not products .

He was caught in a dilemma .

Productization means universality. But universality means a lack of precision .

Customization means precision. But customization means it cannot be scaled up .

something that was both universal and precise .

This is a contradiction .

unless--

Unless the product can learn on its own .

He opened his browser and searched for a word :

Machine Learning " Automated suggestions "

One article title in the search results caught his eye :

“How to Build a Recommendation Engine for Business Decisions”

He clicked on it .

The article says that you can train a model using historical data to learn " what advice is effective in what situations " .

Then, when new data comes in, the model will automatically output the most suitable suggestions .

It's not an if-else rule. It's machine learning .

He finished reading the article and leaned back in his chair .

He thought of Sophia— the data scientist at the company who worked on AI models. He thought of the model with 92% accuracy, and he thought of the data quality issues .

He muttered to himself :

" Perhaps this time, I can do better. "

Chapter 19: Conversations with AI

2024 , Lin Chuan's life underwent two changes .

First, DataPulse gained fifty paying customers and its monthly revenue exceeded $10,000 .

Second, a new tool called ChatGPT has emerged. No, not ChatGPT— it's something newer . An AI that can write code, analyze data, and even provide business advice .

Lin Chuan first used it , his hands were trembling .

He posed a question to it :

" My e-commerce customer repurchase rate is only 15% , how can I improve it? "

Fifteen seconds later, the AI provided its answer: eight suggestions. Each suggestion included an explanation, data support, and implementation steps .

He read it once. Then he read it again .

She writes better than him .

It's not " almost the same " . It's " better " .

He began testing it .

He gave it real data from a DataPulse customer — anonymized — and let it analyze it .

Three minutes later, the AI generated a report .

· He pointed out three anomalies he hadn't anticipated.

· He gave two suggestions that he hadn't expected.

· I even wrote a piece of Python code that can automate one of the suggestions.

He sat there, staring at the screen .

His hands stopped shaking .

His hands were cold .

He sent Daniel a message : " Have you used that new AI ? "

Daniel replied : " I used it. It was horrible. "

you think it 's better than us ?

" In some things, they are ten thousand times better than us . "

" Then what value do we have? "

Daniel didn't reply immediately. Five minutes later, he sent a message :

AI can calculate, but it cannot take responsibility. AI can predict, but it cannot choose. AI can offer advice, but it cannot be accountable for the outcome .

You give a customer advice, and if it's wrong, the customer comes back to you. AI gives a customer advice, and if it's wrong, who does the customer contact ?

A person's value lies not in how fast they can calculate , but in their willingness to take responsibility .

Lin Chuan read it five times .

The next day, he conducted an experiment .

He selected ten of DataPulse 's clients and gave their data to both the AI and himself .

He spent a week writing a customized report for each client .

The AI generated ten reports in three minutes .

He mixed the AI 's report with his own and sent it to a trusted friend — someone with fifteen years of e-commerce experience. He asked his friend to rate it without telling him who wrote it .

The results are in .

AI 's average score: 8.2 .

Lin Chuan's average score: 7.9 .

AI won .

Lin Chuan sat by the window , looking at the Toronto night sky .

The lights of CN Tower were still on. People were still walking on the street. The city didn't stop because of a data engineer 's setback .

He remembered three years ago .

Three years ago, he received his 127th rejection letter and felt that the world didn't need him .

Three years later, an AI told his client that it could do better than him .

The world still doesn't need him .

But this time, he didn't cry. He didn't suffer from insomnia. He didn't think about giving up .

Because he learned something .

He opened his notebook and wrote down the last paragraph :

AI can analyze data, but it cannot understand pain .

When Sarah from the yoga studio said, " I spent three months creating courses, and nobody bought them , " the AI could offer her suggestions, but it couldn't sense her anxiety .

When a pet supply store owner says, " I 'm worried I won't be able to pay the rent next month , " AI can make a prediction for him, but it can't alleviate his fear .

When Mike, the owner of an outdoor gear store, says , " I want to leave this store to my son, " AI can offer him strategies, but it cannot understand his wishes .

Data is cold. Code is dead. Models are unforgiving .

But people are warm-hearted. Choices are flexible. Responsibilities are heavy .

This is the value of a person .

It's not that they're smarter. It's that they're braver .

Chapter Twenty: The Value of Human Beings

Six months later, Lin Chuan stood in a small office in downtown Toronto , looking out the window .

This wasn't his rented apartment. This was his first rented office. It wasn't big, only twenty square meters, on the third floor of an old building on Queen Street . He could see a street full of maple trees outside the window. It was autumn, and the leaves were starting to turn red .

DataPulse had 120 paying customers. He hired two people: an engineer and a customer success manager. The company was no longer just him .

But that 's not why he was standing by the window .

He stood by the window because he had made a decision that morning .

DataPulse 's user backend shows that one customer's account has not been logged in for three consecutive months .

According to procedure, he should send an email saying : " Your account has been inactive for an extended period. Would you like to consider unsubscribing? "

But he didn't send it .

He checked the client's public information. It was a small bookstore in Toronto's west end, which had been open for twelve years. The owner was a gentleman in his sixties .

He then checked the bookstore's sales data —which was automatically collected by DataPulse .

Sales have dropped by 40% in the past six months .

He thought about it .

Then he picked up the phone and dialed the number .

" Hello, this is Book Nook . "

" Hello, this is Lin Chuan from DataPulse . I noticed your account hasn't been logged in for a long time. I was wondering if you've encountered any problems? "

There was a few seconds of silence on the other end of the phone .

" Young man , " the old man said , " my shop may have to close down. "

" Why? "

" Business is bad. People aren't buying books anymore. And I don't know anything about your stuff ... data analysis. I'm just a bookseller. "

Lin Chuan didn't say, " Our tools can help you . " He didn't say, " Let me do an analysis for you . " He didn't say, " You should try this function . "

He said , " Can I come over tomorrow afternoon to take a look? "

you doing here ?

" I don't know. But I'm here, maybe I can help in some way. "

The next afternoon, Lin Chuan went to that bookstore .

It's located on a small street, nestled between a coffee shop and a barbershop . The storefront is small, with a few new books displayed in the window, covered in a thin layer of dust .

He pushed the door open and went in. The bell rang once .

The store was quiet. The only sound was the clicking of the radiators . The bookshelves were very tall, reaching all the way to the ceiling. There was a smell of old paper in the air .

The old gentleman stood up from behind the counter. His hair was completely white, and he wore reading glasses .

" You're that ... Lin Chuan? "

" right. "

They sat down. The old man poured him a cup of tea. It was Lipton, brewed with a tea bag, and it was a little bitter .

“ Tell me , ” Lin Chuan said , “ what do you think the problem is? ”

The old man sighed .

" Young people aren't coming anymore. They all buy books online. Amazon delivers in two days and is cheaper than us. I buy books for $15 , and Amazon sells them for $12 . How can I compete? "

Lin Chuan nodded .

" So, do you want to close the shop ? "

The old man remained silent for a long time .

" I don't want to. I've been here for twelve years. This is my life. But I don't know how to save it. "

Lin Chuan stayed in the bookstore for three hours .

He looked at the bookshelves. He looked at the POS system. He looked at the customers — in three hours , seven people had come in, two bought books, and five looked around and left .

He asked the old man many questions .

· Which books sell best ?

· Which books never get bought ?

· Do you have a membership card ?

· Are there any promotions or events ?

· Are there any online channels ?

· the most loyal customers ?

The old gentleman answered them one by one .

Back home, Lin Chuan turned on his computer .

He didn't use DataPulse . He didn't use AI . He didn't use any automation tools .

He used SQL to query the bookstore's sales data. He created several charts in Excel . He also wrote a report by hand .

It wasn't because he couldn't use tools. It was because he wanted to do it himself .

Because he wanted to remember that data work is not about faster speeds, more accurate models, or higher efficiency .

It was because an old gentleman didn't want to close his bookstore, which he had run for twelve years .

Three days later, he returned to the bookstore with the report .

“ I’ve reviewed your data , ” he said . “ There are three findings. ”

The old man put on his glasses and leaned closer to look .

" First, your core customers aren't young people. They're middle-aged people over forty. They buy novels and history books. This group has a high repurchase rate, but the volume is decreasing. You need to find new ways to reach them. "

" Secondly, you have a corner in your store that sells children's picture books. Sales are very low. But there are three primary schools nearby. If you could partner with the schools to organize reading activities, this corner could become a gateway to your business. "

" Third, your online sales are practically zero. This isn't about opening an online store to compete with Amazon. It's about using social media — like Instagram— to post three updates a week about the bookstore's daily life. Don't sell books, just tell stories. Let people know there's an old bookstore here, an old gentleman, and lots of interesting books. "

The old man listened to the whole thing .

" Are these ... useful ? "

“ I don’t know , ” Lin Chuan said , “ but we can try. ”

They tried it .

Lin Chuan helped the old man open an Instagram account. He filmed the first video: the old man stood in front of a bookshelf, introducing one of his favorite books. The video was short, forty-five seconds long. It was poorly filmed; the lighting was bad, and the sound recording was also poor .

But that video got four thousand views and over two hundred likes. Someone in the comments said , " This bookstore is right next to my house! I'm going tomorrow! "

The following week, the old gentleman held his first reading event. He invited a local author to read picture books after school. Fifteen children and over twenty parents attended .

That day, forty-seven children's picture books were sold .

In the third week, the old gentleman sent an email to his regular customers — written by Lin Chuan. The email contained no promotions, no discounts. Only one sentence :

" This bookstore has been open for twelve years, thank you for still being there. "

that week were 30% higher than the previous week .

A month later, Lin Chuan went to the bookstore again .

The old man stood behind the counter, his expression different from last time .

“ Young man , ” he said , “ I didn’t lose money this month. ”

Lin Chuan smiled .

“ Thank you , ” the old man said , “ you saved my shop. ”

Lin Chuan shook his head .

" I didn't save you. You saved yourself. I just helped you see the data. "

He walked out of the bookstore and stood on that little street .

The maple leaves have turned red. A gust of wind blows, and a few leaves drift down and land on the sidewalk .

He took out his phone and saw a message .

It was sent by Daniel .

" I heard you helped out at a bookstore? "

Lin Chuan replied : " Mm. "

" free ? "

" free . "

" Why? "

Lin Chuan thought for a moment and typed a few words :

Because a person's value cannot be measured by money .

That night, Lin Chuan did not turn on his computer .

He walked to a small park near the CN Tower and sat on a bench .

The Toronto night sky wasn't too dark; the city lights painted the sky a deep blue. In the distance, the lights of an airplane moved like a slow-moving shooting star .

He thought back to the past three years .

127 rejection letters. A Toronto winter. Self-study in a coffee shop. The excitement of running SQL for the first time . The panic of a pipeline failure. The fear of being replaced. Layoff lists. The unease of freelancing. The birth of a product. The impact of AI .

And — that bookstore .

He finally understood .

a data engineer is not data. It's not engineering .

It is a person .

The real starting point and the ultimate goal are the human element .

He stood up and walked back .

Shops on Queen Street were closing one after another. One pizzeria was still lit up, with a few workers who had just finished their shifts sitting inside. A homeless man sat on a street corner, hugging a sleeping bag. A couple walked by hand in hand, the girl smiling .

Lin Chuan walked back to his office and turned on the light .

There was a blank notebook on the table . He opened it to the first page and wrote the first sentence of the book :

" People are not data points. "

He paused for a moment .

Then I wrote the second sentence :

" But data can help people. "

He paused again .

He wrote the third sentence :

" That's why I did this. "

He closed his notebook .

Turn off the lights .

Step out of the office .

The doorbell rang once .

Queen Street , carrying the coolness of autumn and the scent of maple leaves .

He walked into the night .

There was no turning back .

Because the story is not over .

But he is no longer lost .

Epilogue: One year later

DataPulse 's customer base has grown to 300 .

Lin Chuan hired five people. The office moved to a larger place. But every Wednesday afternoon he would still turn off his computer and go to the bookstore to sit for a while .

The old gentleman reserved a special seat for him — by the window, where the sunlight was good and he could see the maple trees on the street .

The bookstore survived. Sales were 45% higher than last year . It gained 8,000 followers on Instagram . Weekly book events were packed .

One day, the old man said to him :

" Lin Chuan, I have an idea. "

" What are your thoughts? "

" I want to set up a corner in the bookstore to display books on data. I'll name it ... ' Data and People ' . "

Lin Chuan smiled .

" good. "

He walked out of the bookstore, the sunlight shining on his face .

My phone vibrated .

It was a text message from an unknown number :

" Mr. Lin Chuan, I own a small bakery. My question is simple: why don't my croissants sell as well as the ones across the street? Could you please take a look? "

He smiled .

Put your phone in your pocket .

I stepped into the sunlight .

(End of article )

📘 Author's Postscript

The novel has twenty chapters, following the life of an unemployed statistics master's graduate, an independent data engineer, and finally, someone who finds the answer .

Lin Chuan's story is a microcosm of many data professionals .

If you're reading this book, you might be experiencing similar confusion: Why can't I find a job after learning so much? Why don't I feel any value in what I've done? With AI coming, what use am I ?

What I want to tell you is :

Data is cold. But you can warm it up .

Code is dead. But you can bring it to life .

Models are unforgiving. But you can make them serve people .

a data engineer is not a path from " not knowing " to " knowing " .

path from " explaining the world " to " changing the world " and then to " understanding people " .

May you not get lost on this journey .

Toronto , Fall 2024

Appendix : Glossary of Technical Terms

the term	explain
SQL	Structured Query Language (SCL) is used for managing and querying relational databases.
ETL	Extract, Transform, Load — the three steps of a data pipeline.
Data warehouse	Data storage system specifically designed for analysis and reporting
Window functions	SQL functions for performing calculations on a set of related rows.
CTE	Common Table Expressions are used to simplify complex queries .
API	Application Programming Interface
Kafka	Distributed message queue system for processing real-time data streams
Airflow	Workflow scheduling platform for orchestrating data pipelines
dbt	Data building tools are used for transformation within a data warehouse .
Spark	Distributed computing engines for processing large-scale data
Data skew	distributed computing, some nodes process far more data than other nodes.
Slowly changing dimensions	Methods for handling changes in dimensional attributes in a data warehouse

📘 Book Summary

The Journey of a Data Engineer is a realistic, introspective, and technically grounded novel that traces the transformation of Lin Chuan, a statistics graduate in Toronto, from an unemployed job seeker to a capable data engineer shaping real-world systems.

The story begins with repeated rejection and a harsh realization: academic excellence does not translate into employability. Through a painful but awakening process, Lin discovers the gap between theoretical knowledge and practical value. This realization leads him into the world of data engineering—a domain focused not on abstract models, but on building the infrastructure that makes data usable.

As Lin progresses, the book explores:

The illusion of “clean data” in academia vs. messy reality in industry
The shift from analysis to system-building
The threat of automation and job displacement
The importance of data quality, pipelines, and reliability
The human side of tech work: anxiety, growth, and identity

Through real-world scenarios—debugging pipelines, handling system failures, building ETL workflows, and supporting AI projects—the novel reveals a deeper truth:

The real value in the data world lies not in explaining data, but in making data usable.

Ultimately, this is not just a technical journey, but a philosophical one—about finding relevance in a system-driven world, and evolving from a “knowledge holder” into a “value creator.”

搜索此博客

Voice of China

Journey of a Data Engineer

📑 Table of Contents (Publisher-Ready)

Preface

Part I — Lost (Reality Shock)

Part II — Awakening (Skills and Cognition)

Part III — Restructuring (Entering the Core Layer)

Part IV — Evolution (System Thinking and Identity)

✍️ Preface (Publisher Style)

📘 Book Summary

评论

发表评论

此博客中的热门博文

特朗普取消出生公民权，能成功吗？Will Trump succeed in abolishing birthright citizenship?

The Ultimate Tool Stack for AI Agents

弦线驻波