Following up from the previous post, "What I learned from Open Compute Project"

I initially thought other clients should buy directly from ODMs instead of OEMs. I later realised the situation is more complex - there's money on the table for a reason. As for cloud versus on-premise, I am rather confused. Both sides claim cost advantages, and it really depends on the scale and context. When I spoke with the hedge fund industry, most opted for on-prem for security and control. What is the cost of compute is a frequent question, but a complex one to answer. The peak flops, the utilisation rate, cost of hardware depreciation, and the desired throughput are some examples of the factors. 

Some of the key insights I've learned are from Prag Mishra: 
1. Flexibility: The cost of FLOP/$ doubles every two years for ML/AI-specific GPUs (source: Epoch AI). Cloud instance requires a fixed-year contract, which might not be cost-effective. By maintaining flexibility to upgrade compute resources, you can cut the cost by half in a year.

 
2. Utilization: 100% utilization in cloud are rare. The max compute per dollar for cloud instance is 25PFlops/$ on a one year upfront. An on-prem GPU server, if utilized 60% of the time, can be a fully deprecated in two years to match the maximum PFLOP/$ that can be achieved in cloud instances. 

3. Execution: On-prem is hard and risky. One missing piece - a generator stuck in backlog - can halt everything. Key components like HBM supply can't scale overnight. At the application layer, OpenAI needs compute that's ready and reliable to serve the customers at speed. 

4. Cost of Performance: In July 2024, the cost of H100 instances in the cloud was $77/hr for AWS p5.48xlarge with 8xH100s. In August 2025, its $55/h - much lower. Actual costs vary depending on the region. 

I've read many resources online for TCO costs, such as semi-analysis or Neoclouds themselves. I'm wary of some of the assumptions, especially the high utilisation rate, which might not be reflective of reality. 

Example: OpenAI Training Estimates 
According to reports from the Information, the New York Times and Epoch AI. For GPT 4.5, Epoch AI estimated that OpenAI utilises between 40,000 to 100,000 H100s to train its model, between 90 to 165 days for $2 per H100 hour. Based on these assumptions and a maximum utilization rate of 90%, the total training cost for a single model is estimated to range between $192 million and $890 million. According to Winsome, OpenAI plans to spend $350B by 2030 on compute infrastructure, with annual server bills at $85B. 

How would you split between on-prem vs cloud?

Decision will be made mainly on technical strengths and costs. Beware of online commentary on clouds. I met an ex-OpenAI technical staff member who spoke highly of his 2023 experience with Azure vs the other cloud provider. I'm cautious about online commentary - much of it reflects incentives rather than reality. From my experience, GPU memory and networking performance often outweigh FLOPs in selecting hardware. This makes Nvidia networking superior for training, and the ecosystem is working hard to catch up - Broadcom recently announced an 800GB Ethernet NIC in a bid against Nvidia ConnectX. 

CXL memory is another industry solution for memory constraints. After talking to Marvell in OCP, my understanding is that it's mostly used by hyperscalers. The software stack is still immature. 

Assuming training occurs on bare-metal nodes, differences in hyperscaler software stacks are largely negligible. What truly matters is speed, reliability and efficiency.


 

Around 2023, I was invited to a dinner party with a bunch of young traders from a particular quant hedgefund. Almost immediately, I was greeted with a socratic debate with a stranger (let's name him John). Enamored by his intelligence, I realized I haven't had an interesting conversation rooted in ideas in years. I began to question what I was doing with my life. Inspired by that energy, I decided I wanted to work with these people.

The fund was elite, with less than 0.1% acceptance rate. I had studied accounting and finance from a non-target school. On paper, it was impossible. But nothing is impossible when humans are involved.

I took a huge bet, knowing that the fund was open to those with math talent. I pushed my “Top in the world mathematics award”. I  knew John was impressed by my intelligence and my story of sneaking into a conference at 19 to find a trading job. I spent 300 hours of my weekend, on top of my 55hour work week, grinding through math. I loved every minute of it. Few months later, I sent "John" a video: a theatrical pouring out of my filled notebooks. I told him I was applying, that I had nothing to lose, and that I'd cold-call his Head of Trading if I didn't get an interview. 

I eventually got the interview and learned that John fought for me in the background. But I failed. The other candidates were simply better. My heart broke, and I walked away with a new mandate: In whatever I do next, I need a 10x competitive advantage.

I've watched AlphaGo as a university student, and was deeply inspired by Demis dream of solving intelligence. AI was taking off. The fund was investing in Coreweave. Other quants were heavily sponsoring ICML. My intuition told me AI had already worked in trading.
When I told a senior trader that XTX Markets had spent $1 billion on a data center, he laughed and called it a bubble.

I suspected he was wrong. I cold-called several CEOs whose numbers I found on Bloomberg. They were loosely involved in AI infra (Gerko,Thiel,Gray). Their assistants politely rejected me, but it only clarified my resolve. I had to go all in. 

I stayed a bit longer for my boss, but asked for one month to step away from my regular work and experiment with building a CNN based trading model. This was completely unheard of on the trading floor - a sales who studied finance working on pytorch code. I failed my experiment. But that showed me how great my boss is, my personal reputation and my convincing abilities. My honesty about quitting hurt my bonus, but it really doesn’t matter. In the long run, you should always leave abit of money in the table. 

A few months after I left, I found this obscure firm writing detailed research on AI Infra for finance. I looked into their securities filing and found out that this quiet company of 10 people is a $100M USD revenue-generating business. I cold-called the company. The other guy was a young junior who was more than happy to exchange knowledge about the opaque business. You never know how much you can learn just by cold calling.

I saw an opportunity to build a business against the incumbents - supplying AI infra for hedgefunds. I knew my edge is in distribution and I was very excited to learn about the product. AI infra is also a horizontal layer technology, which means I can be anywhere in the world in the long term. If i fail in that idea, I can use my knowledge for a job in an AI Lab. This was the thesis I made with the limited information I had.

In Aug 2025, I cold emailed the innovation bureau of HongKong, showing them my blog and my interests. They were excited that someone in Hong Kong knew something related to neural networks and immediately gave me a phone number. It was the director of the local GPU lab. I cold called her and again, like many others, she was amused. She told me to just come and told her juniors to “teach Tiffany something”. I spent one month absorbing all the knowledge from engineers about AI infra supply chain, data centers and testing Chinese GPUs. The feeling of hustling sometimes gives me a sense of thrill - from bending the rules a little, from finding creative ways to get what you want and make things work. 

There were tons of rejections. You just have to keep on trying, move on and forget about it. I’ve been doing this since I was 15. When people told me I was brave for quitting my high paying job, I am not sure if I feel brave. I am just quite used to dealing with uncertainty. I suspect courage is easy when you have no choice but to act. Prior to my success with AISC HK, I spent two weeks in the US physically cold mailing and fedex-ing to multiple CEOs and companies. For one of the company - I printed a t-shirt with a logo of theirs, videoed myself and sent it to every single board member. It’s not just chutzpah, you have to be different, thoughtful and prepared. I got an interview within hours, they were amused but didn’t want to sponsor my visa. In every call, I would always learn something and use that information to think for myself and roll the next dice. 

I became more serious with my business idea and reached out to multiple conferences asking if they need a student volunteer. Open Compute came back with an offer to pay for my flight and hotel as part of charity. I was lucky, once again. I showed up, talk to people and gave them my wild duck PDF. This time, I actually had people proactively reaching out to me and saying they want to help me, offer me a job, none more enthusiastically than a well connected CEO called Dean N. I rejected because I knew it wasn’t the right fit

On the very last day of the conference, I got extremely lucky. I was unpacking my bags, taking out my GEB book on the table, when two older men noticed my “energy”and asked me what’s that book about. I yapped about Kurt Godel for 10 minutes, energetically talked about how that book inspired AI research (I know you might disagree).Turns out they were both senior data center leaders from Intel. 

Over the next 40 minutes, they rigorously challenged my idea. I learned about hidden business practices no one outside could possibly know. This business is far more opaque than I thought. The previous 19 people i met either said my idea would work or said nothing. I also learned about the dynamics of AI Infra that makes margin incredibly difficult. I gave up the idea the next day - I will never be able to make a 10x product. 

At NeurIPS, I  felt aligned. "This is my people", I thought to myself. I was most impressed with a researcher in Reflection AI and two researchers from GDM. Their energy is palpable, and my instinct told me that I can really trust these guys, that I can work with them. 

This is a room with 10,000 of the smartest people in the world. Once again, my heart wants to do something that is statistically difficult to achieve yet gives me so much energy. I looked back at my finance career. All I can think of is the people, not the money, not the fancy food, just the people. 

The math might not always add up, but the chemistry surely did. 

There are broadly two categories of knowledge - a priori and a posteriori. A priori is knowledge from reasoning. A posteriori is knowledge from sensory observation. When people talk, I suspect most of their knowledge comes from a posteriori. As a result, we have the old adage that people don't remember what you say, but how you make them feel. 

Yet few of us spend a lot of time thinking deeply about a priori - trying to understand fundamental truths. I suspect that in most human experiences, there is no such thing as fundamental truth. But we can still break open the deep-seated beliefs that drive our actions and thought processes. 

For the longest time, I believed in the importance of talking simpler. You hear this from Feynman, from Dimon, from people in all walks of life . Yet as a result of talking simpler, I suspect I started thinking simpler. It made me overdeterministic in my thought process, finding plausible reasons behind my actions, my past success and failures. It wasn’t until reading The Brothers Karamazov that I started to deeply question myself—unravelling the complexity of human biases, noise, and the contradictions that exist in our minds.

The shift that followed was fundamental. I now try to speak simpler, but think more complexly. I try to hold two opposing thoughts in my mind at once. At the end of the day, to do well is to articulate well. But articulating well has to be in conjunction with thinking hard.

Writing this blog to demonstrate my understanding of GTM, which is a tad different from macro sales. The big picture, however, hasn't changed; it's all about solving customers' problems and how you make them feel. 

Traditionally, there are two kinds of sales: product-led growth and traditional enterprise sales. I suspect that for ReflectionAI, the latter is more important. The two questions one must ask: Who do you sell it to, What are you selling, and How are you selling?

1. Figure out the Persona 
In PLG, no developer wants to be on the phone, get cold-called, or even pick up a phone call from a random salesperson. In the latter, focus on which persona you want to sell it to. Does your product really resonate with them and drive amazing, incredible value? What is the actual business value, and how do you articulate it? 

Are you aiming for the CIO or some functional leader, like VP of Research? Is it through a referral? Is it cold-outbounding? Some personas you can cold-mail, some personas you can't. 

2. Focus on learnings and insights

Something a junior salesperson can do is thoughtfully reach out to people and focus on gaining insights and logos, rather than immediate deals. No commission targets are probably what makes sense at the start, in order to create a cohesive culture of trust. For an early-stage startup, setting up a commission too early can either be an overtarget or an undertarget. Ultimately, the first 20 hires will be the lifeblood of the culture, who will represent your team as you grow the company worldwide.  

3. Showcase Pilot for customers, focus on executive buy-in 
Make sure executive buy-in is present in pilots. Focus on clear ROI and Results. What does good look like, and how do you measure it? 

While LLMs often deliver noticeable productivity gains, these benefits can feel intangible - so it's critical to define and communicate clear results. 

4. Large Enterprise is moving faster than you think 
There is now pressure from boards on "What is your AI strategy?". When market share is at stake, large enterprises are moving with surprising velocity. Fear of revenue erosion is a powerful accelerant.



“For me, to remember friendship is to recall conversations that it seemed a sin to break off: the ones that made the sacrifice of the following day a trivial one” - Hitchens

During my break in 2025, I did the things I never had the time and money for: travelling cheaply across Asia, learning to ski and pursuing my AI interests.

Yet, for all the excitement of 2025, nothing beats meeting Isaac* and Albert*. We spent countless hours lost in philosophy, history, politics, literature and science, followed by absurd jokes and complete nonsense. 

A lot of friendships and connections depend upon a sort of shared language, not necessarily designed to exclude others, but to instantly bridge the gaps left by time. With Isaac, we would bond via slagging - a high-powered version of teasing where friends are jokingly cruel to each other. I would endlessly tease Isaac’s gigantic forehead and receding hairline, only for him to fire back at my own hairline and the pretentiousness of my latest intellectual comparison. 

"How dare you, Tiff," he’d scoff after I tried to compare Shapiro to Hitchens. "You know absolutely nothing about Hitchens."

It is indeed true - I knew nothing about Hitchens beyond his identity as a debater. But I was quickly awed by his wit, intellect and sheer courage of independent thought. If one has to wince at one’s stupidity, I find a bit of relief in reminding myself of a man who once defaced a political poster in the Middle East with a four-letter word, only to realise too late that it was for a martyr. Hitchens nearly died because of his ignorance and ballsy defiance. I can’t help but be amused by his many adventures and reckless love for life.

Hitchens is one of the many “characters” my friends introduced to me. I am so grateful to have finally found friends who are much more well-read than I am. There is no greater joy than to have a good conversation where wide reading and original ideas finally meet.

Isaac

Isaac is simply very rare. He is one of the few people I know who goes to certain lengths to protect his mind, although I suspect most of it is his natural proclivity. I was positively shell-shocked to learn that he spent his entire life, as a GenZ, never touching social media. No Facebook, no Twitter, not even LinkedIn. Isaac, to me, has spent his entire life protecting the independence of his mind.

You can really feel that discipline in our conversations - his razor-sharp rationale and his effortless erudition. Unsurprisingly, his circle is small. Yet he remains one of the most self assured person I know - perfectly happy in the solitude of his own thoughts. He seemed to have mastered something I often struggle with: the idea that a clear head is a virtue you shouldn’t trade just to escape the weight of being alone.

Conversations with Isaac are always exciting. Isaac has a knack for dismantling my hard questions. Our conversations are often provocative - my questions challenge his logic, and his answers challenge my perspective. Isaac has more than once dismantled my belief systems, and that is no easy feat.

One time, Isaac challenged my idea of “meaning”. I had spent a large portion of my life believing my life was meaningful. By the end of our 8-hour conversation, he had convinced me that there is no real meaning in life - that my sense of meaning is nothing but a fluff of emotions used to justify my own suffering. All along, I was fooling myself, perhaps to make myself feel better. 

Isaac’s intent wasn’t to make me a nihilist, but to ensure that I stopped deceiving myself with narratives. I suppose that is what I loved most about our friendship - always learning, always exciting, always being challenged and the refusal to sugarcoat things.

One of my favourite cheeky memories involves provoking his distaste for Dostoevsky. While I was obsessed with The Brothers Karamazov, Isaac—being a staunch atheist—couldn't stand the religious weight of Crime and Punishment. I took great pleasure in bringing up “Dodo” every chance I got, just to watch him wince.

"If you insist on admiring him so much," he retorted, "the least you could do is learn to pronounce his name correctly." I laughed. In that moment, as always, he was honest to the core. 

Albert 

I met Albert through a series of unlikely circumstances. In mid 2025, I was in the middle of a career "roll the dice" in California, pushing my luck to break into AI. My main motivation was the people; I was desperate to find a circle that was actually inspired by the abstract question of "how the mind works." I wasn't disappointed. Albert shared those motivations, though he was significantly more knowledgeable and—rightfully—a bit older than I was.

I’ve always told my friend John* when I’m about to do something crazy, and he encouraged me to join a chat group started by one of his old Oxbridge connections. It was mostly people yapping about ideas, often with a layer of status-signalling that I didn't take too seriously. But one day, I posted a thread about a meetup for an AI lab, and Albert liked it. I looked him up, saw he actually worked in the sector, and boldly reached out. I wanted to learn about AI infrastructure and sent him my blogs to show my interests. To my surprise, he was amused enough to hop on a call.

We talked about AI for maybe 10 minutes, only to be followed by one of the most engaging (albeit unhinged) conversations I had for a random call with a stranger. For the next two hours, we covered everything from enlightenment, French prepa, science, academia and politics. He was incessantly challenging my conclusions, poking holes in my logic, and we ended the call with him complimenting my “balls”.

Feeling inspired, I followed up with a long, sentimental text about how grateful I was for my life, my family's poor upbringing, and how lucky I felt. I was promptly shell-shocked by his response. He sent back a wall of text explaining how I was absolutely fooling myself into thinking that my entrepreneurial spirit correlates with my upbringing —that I was just telling a story in my head. That I was overdeterministic. 

This was from a stranger I had never even met in person. My heart dropped; I felt like total crap for about ten minutes. But then, a wave of amusement hit me. He was so right. "We are definitely going to be friends," I gleed with excitement.

And I am so glad we did. Albert has become one of my most treasured friendships. He never lets friendship take precedence over his first love, which was and is logic. If one employs flawed thinking, it would be rubbed in; no, it would be emphasized. I suspect that is the very reason he is often accused of “mansplaining”, but it’s the same reason he is so loved dearly by his friends. He cares more about the truth than he does about being polite. He is always interested in how you think rather than in what you think. In a world of superficial small talk, that is a gift.

*Not their real name. Inspired by Camus and Newton.

I’ve spent roughly 8 months gaining deeper knowledge in AI Infra, starting from zero. My initial interest came from an opportunity I saw while I was on the trading floor, and it fits my taste - horizontal layer technology expanding globally, smart people, and revenue growth. I blogged about it, worked hard, read everything, cold-called, learn new information, updated my beliefs, hustled at conferences and learned more. This industry is far more opaque than I thought. 

Earlier in Jan, I eventually connected with a researcher at a leading AI lab who saw something in me. I told him I absolutely have no talent in pioneering new models, who then asked me what I’m interested in. I eventually landed an interview in late Feb with their AI infra team. 

I learned new information during the interview, realised that this particular person is very different from researchers - they are looking for an industry insider rather than high potential, culture fit, values and aptitude. I started thinking about this amazing, young, energetic researcher I met in NeurIPS called Chris and really wanted to be closer to the fulcrum of scientific rigour. Given that I’ve been kamikaze-ing for 8 months, I took a break by reading literature and philosophy and changed my mind about what’s next. 

My fundamental passion is the people I surround myself with - abstract creative thinkers who think about the possibilities they can weave with their technical prowess. It’s time for me to move on from AI Infra and read heavily about open-source LLMs. 

I suppose that's the tough part of being a student again - ultimately, "what" is important is a knowledge held by insider experts, and all you can do is showcase how you think and why, how you think is valuable. 

In my first year of university, I was invited to make a speech at an award ceremony. PT was one of the industry mentors invited to the event - a senior person from a global firm who sat in various government boards in Hong Kong. After my speech, he handed me his namecard, shared his phone number and told me to reach out to him.

Being me, I treated him like a bunch of atoms that decay every 7 years ( a normal human). I was 18 years old, and he was probably around 60 years old. For the next few years, I would visit his office every 3-6 months, and we’d sit for an hour discussing business, life and intellectual topics until he retired. Despite the “me too” climate of those years, he never seemed to care about optics. I think my lack of pretentiousness, persistent questioning and child-like curiosity amuse him. I can’t imagine being that high up and having everyone treat you like a god; it seems exhausting.

I thought it was common sense to treat senior figures with utmost privacy. Afterall, they have something to lose. Turns out it wasn’t common sense.

PT believed in giving back, spending his personal time going to universities in hope of inspiring the next generation. His message was to read more books. I took the advice to heart and started reading. What began as a practical pursuit - gathering knowledge for life - slowly evolved into something deeper: a life of knowledge.

Years later, he said that he began to feel his university visits were futile until he met me. He spoke with frustration about another student mentee who snapped a selfie with him, just to see her showing off on Facebook the next day. He told me he almost gave up on GenZs. When I asked him what kept him going, he simply said: “I am a long-distance marathon runner”.

Having known PT for 7 years now, I could see how this mentorship meant a lot to both of us. For him, he felt a strong sense of fulfilment. To me, it felt like the world cared.

PT, I learned, was not an odd one out. It never ceases to amaze me that the kindness of strangers can be so dependable. As I travelled around the world, moved to a new country and searched for job opportunities, I asked myself, “How will the miracle happen today?”

My first job offer 
Around 2018, I was offered my first internship. How? In the same award ceremony, my accounting professor heard my speech, started talking about it to her colleague, who, I later learned, proactively reached out to multiple friends asking for internship opportunities. A couple of weeks later, my professor asked “Do you want a job?”. I was shocked at the kindness of strangers and said yes.

After I finished that internship, PT called in his head of HR and told her they should hire me as an intern. I didn’t hesitate, I looked at him straight in the eye and told him I didn’t want to do accounting. I wanted  Markets. 

I told him I want to understand how the world works, inspired by the Ray Dalio book he recommended. He laughed. “Tiffany”, he said, “you realize that investment banking has some of the smartest people in the world, right?”

Years later, I learned that it isn’t really true. But at the time, the odds were heavily stacked against me. I was determined anyway.

Why? Because I am driven by a visceral awareness of how short life is. Knowing my mother, my grandfather, and my uncle all faced cancer before the age of fifty, my fifteen-year-old self became convinced that my own clock might stop there, too. (Though, with the trajectory of AI, I hope it changes). That continues to be the driving factor of my life till today, which is why I am so action-oriented. 

The journey

I did everything - I researched online, cold messaged >100 people on linkedin and struck up conversations with peers/leaders in those recruiting events. Most people told me to go to a good grad school. By the end of the recruiting season, I was automatically rejected by most companies and had exactly one interview: Bank of America. Statistically, the odds of landing a role with only one shot are near zero. I knew I had to create my own luck. I had recently read  “The Uses of Adversity”  in  The New Yorker online  . I figured if Sidney Weinberg could go from janitor to Goldman Sachs CEO by knocking on doors during the Gilded Age - a time defined by its staggering wealth gap - I had no excuse to sit still. In 2019, I found a wealth management conference online. I registered using my student email. I suspect they saw "University" and assumed I was a professor. I showed up in W Hotel Hong Kong as a nineteen-year-old. It was technically legal, but we all know I wasn’t supposed to be there. I struck up conversations with a bunch of industry people. When they asked me what I was doing there, I sheepishly told them I wanted to learn about investments. Some immediately assumed I came from wealth and gave me namecards. I was being pitched “secure gold banks” and crypto. It was a nerve-wracking but hilarious memory. 

I reached out to all ten of the name cards I collected and was honest in the email that I was looking for a job in global markets. Through sheer luck, one portfolio manager found my energy contagious, my knowledge proficient, and my audacity amusing. He excitedly shared this encounter with some of his friends, as lunch-time gossip. One of his friends told her husband, who happened to be a head at Goldman Sachs Hong Kong, Mark. I met Mark before my last interview round with Bank of America. He was amused and encouraging, and said, “ If BofA doesn't work out, you’ll always have a place at Goldman." 

That signal of confidence meant everything to a 20-year-old. But life has a cruel way of reinforcing its brevity. Mark was my second closest mentor, after PT. A week before we were set to catch up after my internship, Mark died in a hiking accident, leaving behind a six-month-old son. It was a tragedy I couldn’t have imagined. I never got to properly thank this wonderful American. I promised myself then that I would pay for his son’s education one day. I also promised to thank the people who make an impact while they are still here. That is why I am writing essays.

The Rationality of Hustle
While I rely on luck, I dont fully believe in miracles. My decision to pursue markets was rooted in simple first principles thinking: if I was smart enough to get into Oxford, and banks hire people from Oxford, I didn't suddenly become “stupid” over the course of six months.

Yet, I was consistently told to just accept that the system is rigged. While I accept the things I couldn’t control, I was tenacious about the things I could. Even in the middle of the hustle, I never stopped being grateful for the miracles of my life. If anything, I am driven to do my best because I am aware of the countless people born in a different era who wish they had the opportunities I did - happily unmarried at 25 with a free-spirited Californian attitude and an American accent. I am grateful that I get to move to a high-trust society like Hong Kong, met a boyfriend who loves a passionate woman like me ( would have been unlikely in Indonesia), and built a circle of friendship with AI researchers - all born in different countries, all connected by a shared English language, because we were born in the age of the internet.

In the end, moving through the world with joy—and a fearless, independent streak—has always been my style. But the credit goes to tools that made the world small enough for me to conquer. To that, I can only say THANK YOU!

One of my recent side projects.

Hold up a finger. Could this finger be a different color? Could it be slightly longer? Could it be crooked? But could it be ever be anything other than one finger? The number is obligatory. The number is something the finger essentially has.

Compute is a key lever for AI progress. I wanted to work on the business side of AI Infra, but that itself encompasses many layers - business development, operations, finance, technical program management, etc. Since it really comes down to the right role opening up at the right time, I figured it’s better to build a solid big-picture understanding of how everything fits together, and then go deep on the specifics when the opportunity comes.

Having an entrepreneurial mindset, plus backgrounds in finance and data science, sets me up well for system-level thinking. I think managing costs, hardware/software optimisation, and actually executing are the things that matter most, and I wanted a skill set that lines up with that. I love adventures, and the idea of taking on a role that involves navigating ambiguity in a fast-moving environment really excites me.

The goal of this blog is to cover:

1. High-level aspects of how modern machine learning infrastructure works
2. Hardware advancements that accelerate deep learning workloads 
3. Industry insights. 

It builds on themes from my previous blog posts—including supply chain dynamics, neocloud, TCO analysis, hardware fundamentals, and OCP reflections*:

Machine Learning Infrastructure

A full ML platform usually has two parallel data pipelines: 

• Real-time pipeline: Handles data that arrives continuously and needs low-latency processing
• Batch: Processes large historical datasets on a schedule(hourly, daily, etc) and is usually used for training.

Real-time: Apache Kafka receives continuous events. Events can be a log record, a click event, etc. Flink consumes these events and performs ETL (Extract, Transform and Load) into the real time feature store. Prediction service will use the latest model and features to make instant predictions.

Batch: Data lake stores large volumes of historical data, which is then processed by Spark ETL. The output goes to the Batch Feature Store which is used for training and batch inference, while the features & labels goes to training. Batch prediction jobs periodically run predictions over large datasets, and the output is saved to the data lake. 

Read more »