WHEN THE NEW Macbooks came out a few weeks ago, Nvidia stated that the chips they provided to Apple did not contain the proverbial ‘bad bumps’. Unfortunately for them, an investigation led by The Inquirer proves that not to be the case.

Background
If you recall, Nvidia has been in the spotlight all summer for failing chips due to bad materials and thermal stress. The end result is that bumps, the tiny balls of solder that hold a chip to the green printed circuit board it sits on, crack, and the computer it is in dies. If you want the full technical analysis, read this article (and parts 2 and 3).

Nvidia took a $200 million charge over the problem in July, but the firm refuses to support its customers by saying which parts are defective, and what computers they were sold in. You can get some clue from message boards, with Dell, HP, and Apple being prominent victims.

Nvidia says that the problem only affects notebooks, HP says otherwise. Nvidia assures manufacturers that their machines won’t have problems, manufacturers say otherwise.

In the end, what you have is a massive cover-up that keeps affected customers in the dark. Doing right by them would cost a lot of money, which says a lot about the reason for a cover up. Fixed parts with a new ‘material set’ – basically new bumps and underfill – were phased into production starting in mid-summer, and the old, defective bumps are being sold off slowly alongside the new.

The question of the season is whether or not the brand new Macbook was designed and sold with ‘bad bumps’. Nvidia told us directly that the chips were not using the ‘bad bumps’, and we took their word for it even though internal Nvidia sources were telling us that this was not the case.

One thing to keep in mind however, is that these bumps are so small that they are virtually invisible to the naked eye. In this case, they are about 100 micrometers in diameter, near the diameter of a human hair. To complicate things, they are permanently sandwiched between the chip die and the green fibreglass carrier, the bumps literally solder the two together. They are then covered with an epoxy-like material called underfill.

Nvidia could have shipped chips with bumps made of peanut butter and said that they were gold. As long as the chips functioned, there was almost no way of knowing exactly what they were made of. It is a pretty safe bet for Nvidia to call the parts good publicly, even Apple might not bother to check up on them. Again.

To say definitively what the bumps are made of, you would need to buy a Macbook off the shelf, disassemble it, desolder the chips, saw them in half, encase them in lucite, and run them through a scanning electron microscope equipped with an X-ray microanalysis system like this.

That is exactly what we did.

The Science

Here is the G96 cut in half, ready for analysis

Yes, you read that right, a brand new 15-inch Macbook Pro was purchased in California as soon as they went on sale. This was an off-the-shelf part, not a review sample, not a gift, but a normal model that hundreds of thousands of you bought. It was then secreted to a small lab of mad scientists who do not wish to be named, fearing repercussions from Nvidia and Apple.

These well-meaning boffins took it apart, desoldered the parts, and cut the defenseless notebook into many pieces. With meticulous care, they then ran it through multi-million dollar tools that would tell them exactly what materials the bumps used. Exactly.

This is what most of the Macbook Pro motherboard looks like

The motherboard of the new 15-inch Macbook Pro looks like this (above), with the heat pipes removed. There are basically three chips on it, the Intel CPU on the bottom, the Nvidia MCP79 chipset, and the Nvidia G96 CPU. The MCP79 is marketed under the name 9400M, and the G96 is called a 9600M GT GPU, but we may refer to them as just the 9400 and 9600 in this article.

The bumps have two possibilities, new and old, good and bad respectively. According to Nvidia documentation, the ‘bad bumps’ consist of mostly lead, 95% lead (Pb) in fact, with the remainder being tin (5% Sn). That is why they are called high-lead bumps. The newer ‘good bumps’ are called eutectic, and what they do differently is explained in great detail in the technical links at the top of the article. As far as composition goes, they are about two-thirds tin (63% Sn) and one third lead (37% Pb).

An electron microscope image of the chipset bump with analysis

Here is a closer look at the bump.

An electron micrograph of a 9400 chipset bump

Take a closer look at a portion of the graph that contains the elements in question.

Material analysis of the 9400 bump

Even if you don’t have a degree in material science, you can plainly see that there are two big clumps in the graph. The two-pronged one on the right is tin, and there is notably more of it than there is lead, the spike on the left. This means the bumps on the MCP79/9400 are made of eutectic material (63% Sn, 37% Pb), and they are ‘good’. Nvidia’s story checks out so far.

Take a look at the same data for the 9600.

9600 micrograph and analysis, coloured red for ease of reading

And again, a closeup of the bump.

A good close look at a 9600 bump

And once again a close-up of the graph. (Please note that the original was black and white, we filled it in with red for clarity)

Part of the 9600 analysis, the tail was cropped for readability

Even a communications major can tell that there is one big spike at lead (Pb) and a very small one at tin (Sn). This would fit the profile of high lead (95% Pb, 5% Sn), and is radically different from the ‘good bumps’ of the 9400. The 9600 is unquestionably using ‘bad bumps’, directly contradicting the statements from Nvidia.

If you want more evidence, look at the surfaces of the bumps in the pictures above. Eutectic solder has a melting point that is the same for all components. When it cools, you should get an even physical structure with a fairly consistent grain. That is what you see on the 9400. With non-eutectic solder, the component that cools first should clump, and you can see that on the surface of the 9600.

More Problems
You will notice that we stated in the beginning of the article that Nvidia said the bumps in the chips were good, and you can see from the above data, this is definitely not the case. The computer that ‘donated’ it’s guts for the above analysis was a 15-inch Macbook Pro, purchased off the shelf in California. There was nothing special about it, not a press sample, not even a pre-production version. This is what Nvidia said was good.

How did they say that? Below is the last mail in an email chain between the author and Mike Hara, Vice President of Investor Relations and Communications at Nvidia. Phone numbers and email addresses were removed, and only the formatting was slightly changed for readability. It was sent at 1:52PM CST on October 15, 2008, titled “RE: 9300/9400 materials sets”. The emails appear here in reverse chronological order.

[Begin Email]

oops, sorry.

Michael Hara – NVIDIA Corporation – Vice President of Investor Relations – ( 408) xxx-xxxx – Fax: (408) xxx-xxxx

=======================================================

—–Original Message—–

From: Charlie Demerjian [mailto:charlie@xxxxx]

Sent: Wednesday, October 15, 2008 11:44 AM

To: Michael W Hara

Subject: Re: 9300/9400 materials sets

I assume you mean 9400, not 6400.

Michael W Hara wrote:

> Charlie,

>

> The 9300/6400 and 9600 discrete all use the new material set.

>

>

> Michael Hara – NVIDIA Corporation – Vice President of Investor

Relations

> – (408) xxx-xxxx – Fax: (408) xxx-xxxx

>

> =======================================================

>

> —–Original Message—–

> From: Charlie Demerjian [mailto:charlie@xxxxx]

> Sent: Wednesday, October 15, 2008 9:27 AM

> To: Derek Perez; Michael W Hara

> Subject: 9300/9400 materials sets

>

> Guys,

> The obvious question of the day is, what materials set is used on

> the 9300/9400 and the discrete 9600 used in the macbooks?

>

> -Charlie

[End Email]

As you can see, the question was asked the day after the new Macbooks came out, and it is quite clear in naming all three potential parts, 9300, 9400, and discrete 9600. There is no nuance, and the then brand new Macbooks are directly named. As we have proven above, the statement in that email is simply not true.

One problem that journalists run into every so often is that PR people don’t always tell the truth. They usually do, but every once in a while, they don’t. Sometimes this gets found out, and that inevitably leads to a very embarrassing story, sometimes it doesn’t. The only down side to this for the PR person is their getting caught with their corporate pants down, and the inevitable hit to their reputation.

The problem this time is that Mike Hara is not PR, he is IR, Investor Relations. His main job is to deal with stockholders and analysts, and is consequently under a very different set of rules when speaking to such people. People buying and selling stock expect, and are required to get honest answers to the questions that they ask.

The legal boundaries for IR are the same for talking to analysts, stockholders, the milkman or the press.

Analysis
So, what does this all mean? It suggests that there are 15-inch Macbook Pros being sold with ‘bad bumps’, the same materials that brought down so many HP, Dell and Apple parts, both laptop and desktop. For some odd reason, Nvidia really does not want you to know this.

The first and most obvious question is, does Apple know? Repeated calls to Apple PR were not returned prior to this story, and while that looks pretty damning, it isn’t. Apple will not talk to journalists unless they are assured the response will be fawning, and we do not fit that mold. That said, given the history between Apple and Nvidia, it could go either way.

When we ran our finding past Nvidia prior to publication, Mike Hara replied: “You asked me specifically if the 9400 and 9600 used in the MacBooks were free of all bad bumps. I responded to you that the combination of material underfill and bump is different from the combination that was exhibiting the bump crack field failures earlier in the year.”

We find this problematic. If the bumps were not a problem at all, why were they changed on the 9400? The 9400 is a much cooler-running chip than the 9600, so why change the part that is less likely to die? If the 9600 with ‘good bumps’ is being phased in, why bother with the qualification costs, time, and inventory hassles if it is not a problem?

The other problem comes down to heat. The new Macbooks run hot, very hot. The net is filled with reports of them overheating and hanging. This is most often seen when gaming, a task that stresses the GPUs hard, and results in a ‘black screen of death’. These beasties run extremely hot.

On the surface, the explanation of the Macbooks not getting hot enough to crack the bumps doesn’t stand up. If it is hot enough to sear the flesh off your thigh, it is likely more than able to reach an internal temperature of 60-80C, the point where the underfill softens. If the chips get hot enough to crash, it is unlikely they are running within prescribed thermal boundaries.

If you assume the engineering work was done, and done correctly to keep the 9400 and 9600 in the correct thermal range, why didn’t Nvidia simply say so? Barring a total failure of their lot-tracking system, they had to have known the Macbooks shipped with ‘bad bumps’.

The Future
What do you do from here? At the moment, the simple answer is: avoid the 15-inch Macbook Pro. While there is no assurance that the high-lead bumps will cause a failure, given their history, we cannot recommend that you take the chance.

Apple and Nvidia need to clearly mark which machines have the ‘bad bumps’ so consumers can decide for themselves. Given that Nvidia claims to be transitioning from high-lead to eutectic bumps, it is only a matter of time until the high-lead inventory is depleted, and the Macbooks are safe to buy.

Until that time, you would be well advised to avoid these potentially problematic notebooks. µ

Note
In a statement just before publication, Nvidia’s Mike Hara had the following comment on the situation. “The GeForce 9600 GPU in the MacBook Pro does not have bad bumps. The material set (combination of underfill and bump) that is being used is similar to the material set that has been shipped in 100s of millions of chipsets by the world’s largest semiconductor company.”

I’ve had the new MBP, fully loaded, for 3 days and I’ve made a few observations.

Trackpad: The new trackpad is definitley a cool concept. New gestures, huge real estate and a slicker texture make gliding around almost as much fun as skating on a frozen pond. However, push clicks don’t register as effectivley as they did on the previous model trackpads, and the overall sound of the pad seems clunky, high-pitched and downright cheap by comparison.

The Screen: Back lit LED screen. The new screen is really, really sharp and vivid. I am a graphic designer, and let me tell you, the colours on this sucker pop like you wouldn’t beleive. Is it accurate? Probably not. I’ve been having a tough time calibrating it, but it does look spectacular. The gloss doesn’t seem to bother me too much either. I guess, if I had the choice I would probably ask for a Matte screen version, but… Apple doesn’t care. Quick shout out re: the Hinge issue that’s I’ve seen floating arround the Apple support site. Yes, when your’re lying in bed and have the MBP on your knee at a 45, the lid does close. The hinges aren’t bad by any means, theyre just… different.

The Keyboard: Back lit chicklet style. What can be said? If you like it you like it. If not… Apple doesn’t care. I think it’s pretty cool. The keys seems more spaced out to me but I could be seeing things. Other than that, I would go so far as to say that the plastic used on these keys by comparison to my previous MBP are of a much cheaper grade of plastic. Also, getting dust or hair under them is more of a nuissance than before.

Overall Design: Overall, the 4,1 MBP / Late 2008 unibody Macbook Pro, is really sexy looking. It may take some getting used to but I’m sure in time I will love it more than its predescessor.

I ran a few Xbench tests with noth system side by side. I will post the results when I get some time. I can confidently say that the new Macbook Pro beat its predescessor in every single benchmark. No question.

However, the 320Gb 7200 Hitachi in my new MBP lost horribly to the 320Gb Seagate I bought seperatley and installed myself. Hitachi vs Seagate. Seagate FTW.

I am noticing that the track pad is certainly not as responsive to push clicks as my previous MBP. However, I think this is largely to due the fact that the size of the “click-able” area is so enormous. It does seem that, overall, the new MBP is composed of cheaper grade aluminum and plastic components and does have a clunkier click sound than its predecessor.

In all honesty, I think I like my old (4,1) MBP better then the new (5,1) one. Only time will tell I guess.

When I spoke to both Apple and Nvidia, they denied any knowledge of a faulty GPU and suggested that the problems being experienced by my colleagues and myself were simply coincidence. Today, October 10th, Apple quietly posted Article: TS2377 which states:

In July 2008, NVIDIA publicly acknowledged a higher than normal failure rate for some of their graphics processors due to a packaging defect. At that same time, NVIDIA assured Apple that Mac computers with these graphics processors were not affected. However, after an Apple-led investigation, Apple has determined that some MacBook Pro computers with the NVIDIA GeForce 8600M GT graphics processor may be affected. If the NVIDIA graphics processor in your MacBook Pro has failed, or fails within two years of the original date of purchase, a repair will be done free of charge, even if your MacBook Pro is out of warranty.

Wow! All of the speculation was bang on!? I better be getting a free upgrade on October 14th.

I would return this device, based on so many issues, but unfortunately my carrier, Rogers Wireless, implements a 30 day or 30 minutes policy, and I exceeded my 30 minutes by about 550 in 10 days =(

Step up your game Apple. First, no response regarding the faulty Nvidia GPU in my Macbook Pro and now shotty software upgrades that are actually making my iPhone worse.

it’s a shame. I went from being a lifetime PC/Windows user and a 3 year Blackberry user to a full blown Mac-head. For what? Nothing but headaches.

It just works… my ass.