In the DataCenter – Lynne's Take on Tech

Thoughtful Thursday: Data Center Marauders and SV Villains

Today as our country spins further into self-induced insanity, I thought I’d talk a bit about (drumroll) data centers! This is ordinarily a rather dull topic to most, consisting of equipment and deployments and connectivity, hardware and software and systems, processors and networks, and of course, operating systems.

Since it is a dull but necessary topic that keeps things running, I could not resist writing about them back in the day. I even put together a book for Wiley entitled Inside the Internet Data Center in the early 2000s which predicted many Internet issues from the perspective of the data center administrator and how to deal with them. It was an enjoyable exercise. Unfortunately, the publisher decided it was an uninteresting field of study and canceled the book. I got to keep the advance and soon moved on to other projects. But data centers will always hold a fond place in my technology heart.

Now, everyone is talking about data centers. But not fondly. Not as an intellectual exercise. Nope, they are talking about them as technological marauders, stealing power and water and land from ordinary citizens in small towns. These entities are placed without any consultation or approval of the citizens in their communities. And the deals for taking the land and power and water are too often under nondisclosure, meaning nobody knows until it’s too late.

The big players in AI – Nvidia, Meta, Google, Amazon, OpenAI, and Microsoft – are now seen as the faces of villains. It’s a PR disaster for Silicon Valley. And yet, this is the endgame move. There are lots of pieces and players – and billions of dollars involved.

At the core, a data center is simply a processor receiving a request, accessing a database entry, and sending a result back. The devil is in the details. And there are a lot of devils, ranging from the actual hardware management and maintenance, to connectivity requirements, to database management, update and integrity — not to mention the software and operating systems that superintend these functions.

These operations require power. Lots of power. And they produce heat. Lots of heat. Hence the massive power and water requirements. And the increased demands of AI mean even more power and water for those AI chatterboxes telling folks that they can do no wrong, even though the actual interactivity to the end user is essentially meaningless.

According to McKinsey, almost $7 trillion will be invested in data center acquisition and development by 2030, $3 trillion in real estate and power infrastructure and $4 trillion in “computing hardware infrastructure” ($3.5T in servers and $.8T in storage). This is a mind-boggling number of processors.

Nvidia as the leading vendor of GPUs is beyond excited. At GTC this week, Nvida CEO Jensen Huang hailed these numbers, claiming he sees “at least $1 trillion” in revenue from Nvidia products through 2027. “I am certain computing demand will be much higher than that.” As Motley Fool noted, “This is startling — in a positive way — as it comes only five months after his earlier forecast, suggesting enormous momentum in orders in recent months. Also, Huang’s words imply that, from what he’s seen so far, he thinks revenue could surpass the level of $1 trillion.” If so, McKinsey’s predicted spend is underestimated.“

The real estate side of the equation is equally compelling. Data center REITs are all the rage. While partners in these ventures are often carefully shielded from local scrutiny, huge funds are usually involved in the acquisition of land and construction of essentially a bunch of big concrete shells. Rural locations are targeted. Land is cheaper in rural locations. Political resistance is diluted. And oftentimes water available for agriculture can be recaptured for equipment cooling. In addition, it is easy to buy a few local officials (and relatively cheap) and obtain through strong-arm tactics power easements to the datacenter.

When large corporations come into a community, they usually attempt to co-opt it by selling folks on jobs. However, once the datacenter shells are put together, there really isn’t much of a need for unskilled or semi-skilled labor. A large data center requires very few on-site personnel. Most software and systems administration is done remotely by engineers. Hardware failover means that a lot of CPUs and GPUs can burn out without having to swap out the damaged equipment.

The modern data center isn’t a factory with people working in shifts and products trucked to customers. It’s a factory for analyzing your most personal secrets, foibles, and thoughts, circumscribing every purchase you make and every comment you write, and monetizing it.

If fund managers believe that $4T in real estate is worthwhile, think of what value they place on your personal data. To get a 10-fold investment return, youre looking at the data value of literally hundreds of trillions of dollars estimated. These numbers rival the GDPs of major countries and blocks, with the US, China and the European Union in the $20 trillion economy group. That’s two countries and one economic block.

Now, these numbers are incredible, but also quite ephemeral. The same value of data of citizens incorporated implicitly in GDP as a complex factor is often claimed as a pure monetary value by AI enthusiasts. Nope, this isn’t the ROI. It’s about controlling the mindspace of investors and securing current investments.

What is real and tangible is the resistance of data center construction in communities throughout the US. And that antipathy has gone hand-in-hand with increased public fear of AI used for government surveillance. And most certainly some of the biggest advocates for its use have been companies like Palantir. People don’t like being reminded they’re being watched, even though that’s what the “free Internet” has been all about.

AI advocates have touted AI as the means for companies to find and fire people because they’re no longer needed. Lots of people. Huge numbers of people. This doesn’t make folks worried about their next paycheck feel real good about it. Especially when they open up their power bill and find it’s gone way up because a datacenter in their area has raised costs for everyone.

And finally, the debacles of AI misinformation from overeager AI advisors have become a joke, and a mean one at that. AI for the masses is not a trusted source.

This is how the standing of Silicon Valley companies has plummeted — not in terms of the stock market, but in terms of their trustworthiness.

William and I discussed the benefits of AI and data centers many years ago. Those benefits haven’t changed. But the mishandling, greed, and sheer contempt of major players in Silicon Valley and large investment funds have frittered away the goodwill and obscured the benefits.

It’s too bad. I really do like data center design. I like the software. I like the systems. I like the hardware. I like the benefits of AI.

But I don’t like the companies and pundits. Nobody does.

2020 AMD and Intel: The Grass is Greener on the Other Side of the Chip Business

AMD or Intel? 2020 is the Processor Battleground

AMD, the long neglected stepsister to Intel, has done marvelously well in recent years, primarily due to Intel’s “meltdown” of trust in their flagship processor products and Intel’s delays in shipping new competitive 10nm chips. Coupled with ineffectual senior management and poor board control, Intel, the darling of the Wall Street set, sat wallowing in management paralysis and a moribund stock price until recently.

Meanwhile AMD’s CEO Dr. Lisa Su has been instrumental in moving AMD from its Eeyore approach to life to that of a first-rate competitor in the chip space with its Ryzen, Radeon and Epyc product lines. Dr. Su has not only changed AMD’s attitude – she’s also changed the entire competitive landscape with bold technology moves and strategic partnerships with companies such as Microsoft. Having dealt with the earlier AMD in the 1990’s, where no one would make a decision and the C-suite was filled with ineffectual do-nothings, it has been refreshing to see capable management drive good engineering and product management.

In the last few years, AMD and Intel have swapped places. AMD, the driver in specialist processors, has gone full-bore into the vacuum left by Intel’s strategic blunders and broadened into general processors . Intel, in contrast, has made an old Intel revenue-enhancement approach “new again”, by taking their general processors and specializing them for specific markets.

But believing the grass is greener on the other side of the chip business comes with its consequent perils. Continue reading 2020 AMD and Intel: The Grass is Greener on the Other Side of the Chip Business

Video@Scale: The New Demands are the Old Challenges – Power and Packet Drops

The challenge of creating seamless video experiences on demand has been a long-sought and long-fought dream. FaceBook video@scale brings specialists together to wrestle with the complexity of end-to-end technical tricks and user level satisfaction often at odds.

Lynne Jolitz at FaceBook Video at Scale. (L Jolitz).

The morning was a blitz of corner cases and tightly wound insights. Minutia of transmission of video and complexity, issues of detection of dropped frames in various browser decode, up/down scaling of video quality on-the-fly, issues in CODEC switching, video stream sizing, I-frame synchronization between different video codecs, which codec to use, network versus browser issues (often appear the same), and getting around browser video correction.

But the two items I am going to focus on are the old hard chestnuts: power and packet drops.

Continue reading Video@Scale: The New Demands are the Old Challenges – Power and Packet Drops

The Security Frustrations of Apple’s “Personal” Personal Computer: Device Access, Two Factor ID, and 386BSD Role-Based Security

Recently, a FaceBook friend lamented that he could not access his icloud mail from a device bound to his wife’s icloud access. He also expressed frustration with the security mechanism Apple uses to control access to devices – in particular, two-factor authentication. His annoyance was honest and palpable, but the path to redemption unclear.

Tech people are often blind to the blockers that non-technical people face because we’re used to getting around the problem. Some of these blockers are poorly architected solutions. Others are poorly communicated solutions. All in all, the security frustrations of Apple’s “personal” personal computer are compelling, real and significant. And do merit discussion.

Continue reading The Security Frustrations of Apple’s “Personal” Personal Computer: Device Access, Two Factor ID, and 386BSD Role-Based Security

Google Cloud OnBoard San Francisco: Buried Alive by PR

Lynne and William Jolitz, Google Cloud Onboard, Hilton SF Union Square, July 2017. Image: Jolitz

There are times when a seminar or conference or training session induces trepidation because the expectations are high. One questions whether it was worth the time to travel to the destination, wait to park in the wreck-a-lot, find the coffee urn empty, and then find a chair in the back where you can barely hear the speaker. All the while, slack messages are building up at home base. Is it worth it?

I’ve always found a reason to make the trip worthwhile – a small tidbit of knowledge, an off-the-cuff experience, an interesting speaker. Sometimes I run into an old colleague and we chat over lunch. Maybe even something *new*.

Then there was Google Cloud OnBoard San Francisco. This conference did not meet expectations. And given the stakes in the battle for the cloud between Amazon, IBM and Google, Google must excel. It did not.

Continue reading Google Cloud OnBoard San Francisco: Buried Alive by PR

Delusional mom or out-of-control government agency?

A toddler is snatched by TSA officials from a weeping helpless mom in the middle of a busy airport and wisked away. Nobody helps. Nobody cares. A horror for any parent. But is this story true? Is our civilization so depraved and cowed that government can violate every aspect of decency and not be challenged or even noticed? I suspect many good citizens might agree with this – after all, isn’t government bad?

But of course “who watches the watchers”? There’s nothing like evidence to mess up a good story, and evidence we have. TSA released nine different camera shots of this distraught mom demonstrating *nothing* happened to her or her child. Nothing at all. Sorry folks – nothing to see here. Please remember to pick up your shoes and water bottles on the way out.

The fact that TSA had to release this video footage (long, detailed and from multiple camera angles to mitigate claims of “doctoring”) demonstrates how paranoia dominates, and also why the appearance of airport security for the masses is consequently just as ridiculous as the culture.

When I reviewed one of Schneier’s books on security and culture, I was struck by his observation that security is handled in an “overt” fashion… public searches, obvious cameras, announcements, shoe and lotion inspections, and so forth, to provide the appearance of serious involvement. But many of these “glamurity” measures, while juicing up the public, are not the ones that are likely to uncover the real bad guys – remember that a group of determined terrorists took over planes with box cutters – those little blades to cut open boxes – not AK47’s or switchblades or cologne. It was organization, intimidation and the element of surprise that allowed them succeed.

So the greatest concern regarding security in airports isn’t necessarily inspecting baby bottles (although on the basis that a bomb could be slipped into an unsuspecting child’s backpack or grandma’s purse, *everyone* must be searched – see, there’s that “organization” and “planning” stuff by determined bad guys again). Nope, the smart investment is in areas of automated photo recognition (do I know you?), examination of flight records (frequent flier? holiday to Tuva?), purchasing habits (cash or credit card? one-way or round-trip?) and ID (are you who you say you are and why are you traveling anyway?). This means realtime database analysis (a form of “business intelligence” pioneered by guys like Tandem to track your phone calls and credit cards – we *are* a consumer society after all) and lots of digital cameras. Oh, it also helps to have smart police who use their “instinct” to check out things – even though 9 times out of 10 there’s nothing there, there’s always that “tenth” time…

So what’s the moral of this little story? That bloggers lie to get hits? Well, I think that we already knew that. That some women are crazy? Given the road rage I see daily it’s not just women here, but there’s a thick percentage of “crazies” everywhere. Nope, the moral is pretty simple: You are being recorded, and not just from the cameras you see or the cameras the staff knows about, but also from cameras the staff and you *don’t* know about. This data is *collected* and *analyzed* and can persist and be pulled for review long after you’ve had that “claimed” incident with TSA or the janitor. To be fair, it’s unlikely to be reviewed – after all, millions of people pass through crowded airports and this means petabytes of uncompressed data that has to be stored somewhere so the persistence time is likely short. But since claims must be made quickly in a 24/7 Internet world, anyone who blogs that “TSA stole my lunch” yesterday on my business trip may actually face video surveillance footage that either shows the staff scarfing down fajitas or shows…nothing at all.

But why, you may ask, are there so many cameras? Aren’t one or two enough? Isn’t that a “waste” of taxpayer’s money. Not necessarily, because subverting security is something that insiders like staff are prone to, hence like banks the vast amount of data collection revolves around monitoring the workers with access – did she just go around the gate? did he just feel up the customers? did they steal from the luggage? and so forth.

But as a personal observation, I’d like to point out a common sense analysis that doesn’t rely on technology nor expertise, but only relies on an understanding of human nature. I felt the most unbelievable aspect of this woman’s blogged claim of TSA child abuse was that nobody in line at the airport inspection station noticed or said anything during this “incident”. Now seriously, I know this is a paranoid “fear culture” where “nobody helps nobody but himself” (to paraphrase a con man), but do people really think that the woman waiting behind this distressed mother or the businessman just ahead of her waiting on a laptop inspection or the grandparents three feet away are *not* going to notice something as unusual as an agent taking a toddler away from his weeping mom? That during an unfolding drama people waiting impatiently to get to a plane will not notice the delay, press in closer and begin to demand explanations?

This is why this woman’s posting was complete nonsense – it completely ignores that we are social creatures who always want to know what’s happening with others. We comment. We rant. We watch. We get upset. Just as a couple of chimps arguing over a banana will cause the rest of the troop to press in closer, people will get involved – especially if there is a child. Grandma will crowd in closer to learn what is going on, the businesswoman four feet away will express concern for the toddler, a twenty-something will ask to speak to another agent. It is human nature to meddle in the affairs of others – that’s what being social animals is all about.

America is full of problems we need to solve to avoid a distopian future, and misconduct by those with badges does occur and must be dealt with appropriately. But there are also lots of scammers, liars and jerks who feed off of the paranoia of our society and make it look a hell of a lot worse than it is. These bottom feeders destroy trust, blacken reputations and encourage cynicism. Instead of focusing our energy on solving real problems, we are instead distracted by idiots enamored with celebrity. We waste time. We waste energy. We lose as a society.

So while some might wish to dismiss this incident, I’d like to expand upon it as an object lesson in how going too far to aggrandize oneself can result in serious blowback. And I’d rather see a fame-obsessed woman trying to get a blog audience to raise her google adwords paycheck exposed as a liar and use this lesson to engage in a discussion of real security needs than see the converse – that in a crowded airport nobody would come to the aid or even question essentially the official kidnapping of a toddler. That so many people are still willing to believe the worst here despite evidence to the contrary says everything about trust in our democracy.

Taking a Byte out of Cookies

When I wrote Memories and Cookies for Byte several years ago after the dot-com boom went bust, I got pushback from the editor. Why would anyone care about persistence, monitoring and cookies structures? As a Director at one of those Internet datacenter companies at the height (and fall) of the bubble, I knew that cookies were very important to bizdev and sales as an indicator to tracking unique visitors. Of course, the underlying assumption was that cookies were persistent even though browsers allowed one to selectively delete them. On my modern Firefox browser there is even a special “remove all cookies” button that makes non-flash cookie removal a snap (flash cookies, aka local shared objects in flash-speak, are persistent objects embedded in the flash plug-in, and not removed by the browser’s cookie mechanism; this is one reason lots of sites are going to flash). And remove them we do — up to 1/3 of computer users remove cookies at least once per month, according to comScore, and 7% of computers account for 35% of all cookies served.

While not surprising, this has serious implications for ad monetization.

FilmLoop and Alexa – When Fake Rankings Kill Companies

There’s much sturm und drang about the nasty setup and selloff of a little company called FilmLoop in the business community this week. While there can be much debated about liquidation preferences, drag-along rights and sharp practices, there is one issue relevant to the datacenter operator – the claim by some VCs that Alexa rankings are a good validator of an Internet company’s audience and future traffic. And now, since we’ve stepped out of money-land and into datacenter-land, let’s examine this assumption a bit more carefully. Is Alexa a good validator of a business, or not?

Fun Friday: Google Test Positive, Laser Bits, Gender Blues

While we were working on getting all those Jolix 386BSD fans their Porting Unix to the 386 articles (we have been swamped BTW – and yes, there’s more coming), a few other items of interest this week…

If you made money on Google (or if you wished you had made money on Google), you might try using The Google Test to evaluate your next investment. According to Matt Marshall of VentureBeat, “Entrepreneur William Jolitz posits a contrarian view on YouTube, praising its expensive use of bandwidth as a key to its success. Read on about how YouTube meets the “Google Test.“

Bandwidth-driven business models may seen counter-intuitive to a technologist – after all, we like to make products that make money, and when we fail we tend to face a firing squad. So I can understand it when a techie gets all worked up about those $1M/mo bandwidth bills without getting anything in the way of dollars back. It’s always bothered me that companies like Google, YouTube, and MySpace seem to violate laws of nature. But guess what, kids – making these big Internet moves doesn’t seem to have as much to do with moving $$ product (after all, any techie can upload a movie or make a GenY webpage with a plethera of packages) and more to do with getting eyeballs and mindspace. Yes, this was true when I was at IGN Networks way back in the dotcom bubble, and it’s still true today.

So I heartily recommend that tech guys and gals read this little piece, not to depress you, but to allow you to learn how those top-flight venture firms like Sequoia make their decisions. After all, if they want to spend the money, aren’t we up to the challenge of making it work out?

On another exciting topic, looks like Intel and UC Santa Barbara have made a a promising breakthrough in using laser light to make a much faster interchip switch. According to the article “The breakthrough was achieved by bonding a layer of light-emitting indium phosphide onto the surface of a standard silicon chip etched with special channels that act as light-wave guides. The resulting sandwich has the potential to create on a computer chip hundreds and possibly thousands of tiny, bright lasers that can be switched on and off billions of times a second.” As one of my engineering friends said when we chatted, this makes low-power SiliconTCP all the more valuable (see InterProphet for more information).

Finally, an interesting essay from up-and-coming Renkoo CTO Joyce Park on women role models in business and her dismay over the HP Dunn affair. Since I’ve written on this topic often, I couldn’t help putting in my own two-cents. 🙂

Happy weekend reading – there will be a “Google test” on Monday.

Exploitation of Child Labor in Tech? I Can’t Hear You Over My IPOD…

This spring I interviewed a number of “The Tech” museum award laureates who have used technology to improve the human condition (usually extremely frugally). One of the most exciting innovators is Saeed Awan, director of the Centre for the Improvement of Working Conditions & Environment based in Pakistan. They tackled the problem of child labor in the rug weaving industry. Instead of simply outlawing the practice (which would be futile because these child laborers are the major breadwinners for their poor families), they “engineered out” child labor by developing a “improved, ergonomic and, most important, adult-friendly loom”. Coupled with economic practices (loom ownership allows bidding on lots by adults), this innovation moves the breadwinner status back to adults (primarily women) and moves children back to schools. A perfect use of technology and worthy of a “tech” award.

But where technology providers giveth, technology providers also taketh away…