Copying the text just incase FB decides they don't like it
---
Hey everyone, we're going to be deleting our Facebook page in the next couple of weeks, but we wanted to explain why before we do. A couple months ago, when we were preparing to launch the new Limited Run, we started to experiment with Facebook ads. Unfortunately, while testing their ad system, we noticed some very strange things. Facebook was charging us for clicks, yet we could only verify about 20% of them actually showing up on our site. At first, we thought it was our analytics service. We tried signing up for a handful of other big name companies, and still, we couldn't verify more than 15-20% of clicks. So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on. And if the person clicking the ad doesn't have JavaScript, it's very difficult for an analytics service to verify the click. What's important here is that in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook. So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs. So we tried contacting Facebook about this. Unfortunately, they wouldn't reply. Do we know who the bots belong too? No. Are we accusing Facebook of using bots to drive up advertising revenue. No. Is it strange? Yes. But let's move on, because who the bots belong to isn't provable.
While we were testing Facebook ads, we were also trying to get Facebook to let us change our name, because we're not Limited Pressing anymore. We contacted them on many occasions about this. Finally, we got a call from someone at Facebook. They said they would allow us to change our name. NICE! But only if we agreed to spend $2000 or more in advertising a month. That's correct. Facebook was holding our name hostage. So we did what any good hardcore kids would do. We cursed that piece of shit out! Damn we were so pissed. We still are. This is why we need to delete this page and move away from Facebook. They're scumbags and we just don't have the patience for scumbags.
Thanks to everyone who has supported this page and liked our posts. We really appreciate it. If you'd like to follow us on Twitter, where we don't get shaken down, you can do so here: http://twitter.com/limitedrun
Happend to us too!! we first thought kissmetrics was wrong, so we built a logger that logged server side requests and a client side code that executes on load of the landing page, and we found that 70% of the server side hits had javascript disabled(or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).
> (or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).
That's actually an interesting proposal-- 100% of the time when I click on an ad, I didn't mean to and I try to close it before it has a chance to annoy me.
Probably doesn't make sense on a large scale, but it's something to consider.
If that is true and 70-80 percent hit the back button before the js files load, it still says a lot about the quality of the clicks from the perspective of someone considering advertising on FB...
Someone's grandmother miss-clicking and then mashing back is about as useful to the advertiser as a bot. conversions are where it's at. Poor quality traffic / bots / it's all the same. the advertiser quits paying
I agree that some users might close the page or hit the "back" button before the $(document).ready() happens, but I can't believe that 70% of the people did that(taking into account that our ad was really targeted and highly relevant to the landing page).
It was a few months ago and we've already deleted the db table, we are soon relaunching with a new campaign, so once I get the data again I'll post it here.
I assumed this just meant they built something on top of access logs that told them what percentage of page loads also loaded JavaScript. I don't know about your webserver, but mine doesn't tell me this kind of thing out of the box.
Anyway, it's pretty irrelevant whether they added logging in their app (one line of code?) or enabled their webserver's built in logging.
Access logs that write continually from different processes or threads (like PHP) can cause System IO to reach critical proportions. In any non-trivial enterprise application it's completely reasonable to have a stat-logic-layer for recording events... The access log can still do it's own sorta thing - this is more true if you have more than 1 server.
You wouldn't, but if the vast majority of your ad click visitors have javascript disabled, and only a tiny percentage of your other visitors do, what conclusion would you draw?
You don't, but it's very unlikely that there were enough noscript users to match the stats. From the post:
"...in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook."
For example, target something for the Reddit.com users and you'll see 90%+ block ads.
With facebook granular targetting you could very specifically end up with a target segment with a large proportion of noscript type plugin users. Without more details on their campaign I would not be so quick to place blame purely on bots.
But this is about users who click on ads on Facebook. The sort of users who have javascript disabled are going to self-select out of that group pretty strongly.
And even among the most technically literate groups, I'd be amazed if you saw 80% use of noscript. So many sites require javascript that it's easier to browse with it enabled.
There are a few tricks to do that, described on the old ha.ckers.org weblog, or perhaps on the related sla.ckers.org forum. No idea if they still work, though ..
You could always see if you could trigger NoScript's built-in XSS or frame hijacking protection :) Bots will not notice, and users get a scary warning message and believe your ad is trying to hack them. Not good for your conversion rates, but a pretty sure way to tell them apart.
What you say is valid, however no-script users are a small minority of internet users. I doubt their numbers would tip the results in either direction.
This could be a businessperson's interpretation of what happened.
Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.
Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.
Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).
It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.
It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl.
I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.
You had awk? Ha! Luxuries! Why when I was a young programmer we had to write the code in the
snow with our pee, and a compiler was just a word for the pilot of the
hovering dirigible that read the instructions and passed them to the
ALU, which was another fellow with an abacus. They would wrap the re-
sults around a rock, and drop it on my house when the program would
exit. We had to walk uphill...
Is it though? It seems they spent a lot of time throwing more and more javascript at the problem until they realized they couldn't figure out what was happening. If they had real logging from the beginning, it would have unconfused them much sooner.
They were taking orthogonal measurements to see if the pattern would appear over and over. Also it sounds like each successive measurement technique was at a lower level than the previous technique. Sounds pretty thorough. I'd love to see some actual data though.
Access logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions.
Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for example.
The way I understand this is that they want to log who accesses their pages on FaceBook's servers.
I am not familiar with how things work on FaceBook, but I would think that FaceBook users do not have access to the logs that FaceBook's servers write.
you're not alone. I've heard similar stories from other businesses that have tried to advertise on FB. At first I thought they just didn't track properly, but it seems it's a widespread problem.
And so it seems that Facebook could be proven to be the new groupon.
I would absolutely not be suprised if the hits were FB bots - however I'd expect they wouldn't be that directly stupid so as to have an Ad-bot net in house. Surely this is some service they outsource for plausible deniability.
Something they would have learned from their relationship handing all user data over to the NSA/CIA.
EDIT:
I'd like to make a prediction on how this could potentially play out:
1. FB ignores them. They have nothing to do with this issue and they know the bots are tainting traffic - but they can't/wont do anything about it because it boosts their revenues.
2. FB may or may not be involved in the ad-bots, but, they will drop their $ contingency on ad-rev for the name and payout these guys to shut them up. Their stock was improperly priced and taken a beating. They don't want this coming to light in a large way as it will have a negative affect on the perception of their business model. They hav to be kind of careful of the Barbara Streisand effect here too...
3. Many more people will reveal the same results with empirical testing and it will be revealed that FB earnings are 80% over inflated. Their stock will drop to the predicted real value of <$10... maybe hit that actual projected number of $7.00 - Zynga will BEG FB to purchase them as they are now a penny stock and they can't sustain.
We need to see some real numbers before making these kind of predictions. What was the volume of clicks that OP couldn't detect? 10, 1K, 1M? It makes a big difference.
Also given FB users' aversion to ads this whole issue could be explained by users just clicking the back button. Ad supported sites have trained users to do three things very well: install adblock, habitually ignore ads, quickly click away from any interstitial or unexpected ads like on Hulu.
Another note on the "back button": I recently discovered the awesome power of "three-finger swipe back button" on Macbook trackpad. So it's possible users can "click" back even faster than before. I can and before the swipe I was well-trained in the art of "CMD back arrow" :-)
Totally agree. However, I wanted to put my thoughts on this down... Surely we need more info and time to see what the actual truth is - but I just do not believe that FB is in any way some altruistic innocent/neutral entity.
What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.
Also, FB may not have much of a choice to comply with those agencies, or find themselves pissing off a lot of high level people by refusing to comply with government agencies. But outsourcing / tolerating a network of bots to vastly boost revenue is probably considered fraud, no?
> What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.
It's not what people brag about, it's the social graph itself that's useful. All the other things like photo-tagging are extra candy.
More info at http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/al... and http://nplusonemag.com/leave-your-cellphone-at-home . Sorry it's late out here, or I'd copy/paste the relevant quotes for you--look for the bits about data mining and machine learning, especially in the 2nd article, everything is a signal and they do have the computing power to process it all, apply Bayesian belief nets, algorithms can pull intelligence information about you from the local shape of your social graph that you don't even realize can be inferred from it (humans are not very good at reading complex info from big graphs much further than simple "friend-of-a-friend" relationships).
So without ANY facts your conclusion is that Facebook is fraudulently deceiving advertisers.
And that they are doing this using experience from working with the NSA/CIA in order to cover up the fact that $800 million dollars of Facebook's yearly profits will disappear overnight once people realise the truth. And as such they are willing to pay out the ad-bot companies in order to hide this conspiracy.
I think problem is more deep, I do not want to make other Conspiracy Theory but here some story from other huge Russian social network (vk 150m. users). There is one small, not that big group (like fb groups), users made agreement, that no one will post next day (something like honeypot), and those who post would be considered as bot and baned forever. So next day couple or so bots generated comments encouraging others do not go on strike against some political party [0].
I do not know is it applies to FB, but those who have like 100k bots have some power too.
What about the farking article? Where they proved that 80% of their clicks were from bots. Then several others confirm they ahve the same experience?
What about those facts?
I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.
Am I extremely distrustful of Facebook? Of course! Maybe you have not followed their track record or the character of their founder.
If you are into conspiracy theories then how about maybe this is Google directing a massive bot net to make facebook ads useless?
> I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.
Great argument! A lot of people believe it, so it must be true! Are you for real?!
It is not my responsibility to educate you on modern internet history. This has been argued here on HN prior; Do you recall the backdoors that AT&T put at 600 Folsom, in SF?
And the backdoors in Cisco's equipment that are a requirement by the federal government?
Maybe you thin this shit is "conspiracy theory" and if you do, you're simply a naive fool.
Look around you at what governments are doing. The NSA has records on everything you do online. They had a system in 2005 which could trace communications between users to 6 degrees, automatically.
I know a couple of guys like you in my local activist community, who take a very hostile "I know the truth and you're all fools!" attitude, despite the fact that their audience is mostly very sympathetic to most of their assertions. We know spying on the Internet takes place; HN is full of cantankerous old Internet geeks who've seen, first-hand, plenty of examples of the state (whatever state you may choose, as it happens all over the world) behaving unethically on the Internet, spying on people and punishing people for things that shouldn't be crimes.
But, your paranoid approach is counter-productive. You might as well be working for the people you claim to be afraid of, for all the good you do (negative good; you're convincing people that the folks who believe the government is spying are all paranoid nutjobs who scream at anyone who has the gall to mention other possible explanations).
So, let's review:
1. Just because spying has taken place, and is currently taking place, and may even have the complicity of facebook in that spying, it does not mean that facebook is running a botnet to steal advertiser dollars. The simplest explanation is that facebook looks the other way while others run the botnets. facebook wins (a lot, as long as most advertisers don't know it's happening), botnet owner wins (a little), and the advertiser loses. But, there are other plausible explanations, including incompetence.
2. When you paint things in a "Either you accept my theory in its entirety, or you're all idiots", you force people to choose a side. Nobody wants to be on the same side as an asshole, so you force them to choose the other side. You make people who may even agree with you (to a greater or lesser degree) to begin to formulate plausible reasons for why you're wrong about the crazier stuff you're spouting...further convincing themselves that you're entirely wrong. The best you can hope for is people ignore you and don't have the chance to be inoculated against your ideas; having you as their first exposure to these concepts guarantees they will be less likely to believe them in the future, even if they come from a more credible source. Humans are funny creatures.
Thus, I would point out that there are some really naive and stupid people on HN that have zero grasp of effective argument, persuasion, and even basic logic.
You're right. It was an unproductive method of describing the behavior I was seeing (and was too easy to interpret as saying the person is schizophrenic rather than exhibiting behavior I associate with schizophrenia). It was also insensitive to schizophrenics.
In my defense, schizophrenia runs in my family, and I'm very familiar with it...I don't think of it as an insult. But that's a local custom in my family that I shouldn't think follows in the rest of the world.
>But, your paranoid schizophrenic approach is counter-productive.
Really, now I am a paranoid schizophrenic?
Just because I make claims which are readily confirmed and were completely available in the media - even the EFF filed suit on the AT&T events...
Yet, for some reason, it is my responsibility to educate everyone every single time someone new comes along who hasn't been following these things closely.
Now I am a paranoid schizophrenic?
Also, its a strawman to focus on my tone, rather than content. You're trying really hard to be overly pedantic and, frankly, an asshole.
What I am witnessing is someone trying to inform people, while effectively censoring himself by delivering in such a way as impedes its delivery.
You obviously have a vested interest in seeing your "message" be received, so stop ignoring what people are telling you about your tone and change it. Figure out what it is that people do listen to, or else you have no one but yourself to blame when they take issue with your tone.
To wit: Your tone is defensive. Pejoratives and cursing undermine your expression. What you are effectively telling people is that your thoughts are not important enough to merit self-restraint. Detailed explanations following assertions are also helpful.
FWIW, I work in the performance marketing industry and, IMO, the most complicity facebook could be said to have in this issue is in not placing a sufficiently high priority on preventing bots from clicking on their ads. Even if it is a sizable project there, they represent such a large target that the difficulty of the job becomes much greater. I see no salacious story here, other than a company found itself unable to optimize a campaign into profitability, which I think says more about them than facebook. Ho hum, find a different traffic source and move on.
Thanks - Ill take the constructive criticism on my tone.
However, I will point out that with respect to your comment on FB's complicit actions due to the daunting nature of the problem, this does not take into account their other actions of a 24K ransom on the domain.
Everyone can argue in any direction they want - but neither me nor anyone else is really going to know until we get further down this path...
He's not arguing with you, he can't be using a strawman. He's just pointing out that other people will be distracted by your tone, making them ignore your (mostly correct) message.
And there are some that actually know what they're talking about in terms if intelligence agencies but choose not to indulge in idle conspiracy hypothesizing. The real operations aren't something that the people with actual knowledge would ever discuss. Those that do claim conspiracies are often ill informed or just have read too many spy novels.
>in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled //
I use noscript. When I enable it for visits it's usually by enabling the domain itself and leaving third-party scripts disabled. Depending on your area this could be significant.
Also the possibility of blocking of the scripts by other elements not loading - for example if the browser can't parallelise the requests for some reason and a preceding request can't be handled then the page might be displaying well before the script is loaded, especially if it's loaded at the bottom of the markup.
There is (subjectively) a marked deterioration of comment quality on front-page discussions in the past few days, I feel. I have also noticed that HN articles are getting linked on reddit more frequently (I use both networks, but for highly disparate purposes - gotta get those funny .gif's somewhere). No data supporting this but I fell it's a trend and I'm not liking it.
Few days? Try since forever. As people increase the SNR gets larger. Even on reddit when it started, it was heavily tech focused, you'll find a comment that says it's going downhill. Hard to say what the numbers on HN are like growth wise in comparison to reddit (which has their subscriber numbers on display).
Comments
Copying the text just incase FB decides they don't like it
---
Hey everyone, we're going to be deleting our Facebook page in the next couple of weeks, but we wanted to explain why before we do. A couple months ago, when we were preparing to launch the new Limited Run, we started to experiment with Facebook ads. Unfortunately, while testing their ad system, we noticed some very strange things. Facebook was charging us for clicks, yet we could only verify about 20% of them actually showing up on our site. At first, we thought it was our analytics service. We tried signing up for a handful of other big name companies, and still, we couldn't verify more than 15-20% of clicks. So we did what any good developers would do. We built our own analytic software. Here's what we found: on about 80% of the clicks Facebook was charging us for, JavaScript wasn't on. And if the person clicking the ad doesn't have JavaScript, it's very difficult for an analytics service to verify the click. What's important here is that in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook. So we did what any good developers would do. We built a page logger. Any time a page was loaded, we'd keep track of it. You know what we found? The 80% of clicks we were paying for were from bots. That's correct. Bots were loading pages and driving up our advertising costs. So we tried contacting Facebook about this. Unfortunately, they wouldn't reply. Do we know who the bots belong too? No. Are we accusing Facebook of using bots to drive up advertising revenue. No. Is it strange? Yes. But let's move on, because who the bots belong to isn't provable.
While we were testing Facebook ads, we were also trying to get Facebook to let us change our name, because we're not Limited Pressing anymore. We contacted them on many occasions about this. Finally, we got a call from someone at Facebook. They said they would allow us to change our name. NICE! But only if we agreed to spend $2000 or more in advertising a month. That's correct. Facebook was holding our name hostage. So we did what any good hardcore kids would do. We cursed that piece of shit out! Damn we were so pissed. We still are. This is why we need to delete this page and move away from Facebook. They're scumbags and we just don't have the patience for scumbags.
Thanks to everyone who has supported this page and liked our posts. We really appreciate it. If you'd like to follow us on Twitter, where we don't get shaken down, you can do so here: http://twitter.com/limitedrun
Happend to us too!! we first thought kissmetrics was wrong, so we built a logger that logged server side requests and a client side code that executes on load of the landing page, and we found that 70% of the server side hits had javascript disabled(or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).
> (or that hit the page and left it before the js executed, which seems improbable for such a large number - 70%).
That's actually an interesting proposal-- 100% of the time when I click on an ad, I didn't mean to and I try to close it before it has a chance to annoy me.
Probably doesn't make sense on a large scale, but it's something to consider.
If that is true and 70-80 percent hit the back button before the js files load, it still says a lot about the quality of the clicks from the perspective of someone considering advertising on FB...
so the net result would be the same.
Someone's grandmother miss-clicking and then mashing back is about as useful to the advertiser as a bot. conversions are where it's at. Poor quality traffic / bots / it's all the same. the advertiser quits paying
To be fair, that pretty much matches what I've heard about Facebook's click quality.
I agree that some users might close the page or hit the "back" button before the $(document).ready() happens, but I can't believe that 70% of the people did that(taking into account that our ad was really targeted and highly relevant to the landing page).
and then we find out drunk or stupid people accidentally click stuff.
drunk or stupid people probably won't click 'back' that quick.
What user agent was reported by those hits? This is fascinating.
It was a few months ago and we've already deleted the db table, we are soon relaunching with a new campaign, so once I get the data again I'll post it here.
70% is a ratio ! The effective number may be small at the same time.
We built a page logger. Any time a page was loaded, we'd keep track of it.
They reinvented the access log? By hand? Are people really that detached from how servers work?
I assumed this just meant they built something on top of access logs that told them what percentage of page loads also loaded JavaScript. I don't know about your webserver, but mine doesn't tell me this kind of thing out of the box.
Anyway, it's pretty irrelevant whether they added logging in their app (one line of code?) or enabled their webserver's built in logging.
Access logs that write continually from different processes or threads (like PHP) can cause System IO to reach critical proportions. In any non-trivial enterprise application it's completely reasonable to have a stat-logic-layer for recording events... The access log can still do it's own sorta thing - this is more true if you have more than 1 server.
Surprised no ones mentioned the noscript plugin.
Since the user is on facebook they'd have facebook unblocked, but noscript still blocking everything else.
How would you discern a noscript user from bot?
You wouldn't, but if the vast majority of your ad click visitors have javascript disabled, and only a tiny percentage of your other visitors do, what conclusion would you draw?
You don't, but it's very unlikely that there were enough noscript users to match the stats. From the post:
"...in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled, not 80% like these clicks coming from Facebook."
It's a different marketing block.
For example, target something for the Reddit.com users and you'll see 90%+ block ads.
With facebook granular targetting you could very specifically end up with a target segment with a large proportion of noscript type plugin users. Without more details on their campaign I would not be so quick to place blame purely on bots.
But this is about users who click on ads on Facebook. The sort of users who have javascript disabled are going to self-select out of that group pretty strongly.
And even among the most technically literate groups, I'd be amazed if you saw 80% use of noscript. So many sites require javascript that it's easier to browse with it enabled.
> How would you discern a noscript user from bot?
There are a few tricks to do that, described on the old ha.ckers.org weblog, or perhaps on the related sla.ckers.org forum. No idea if they still work, though ..
You could always see if you could trigger NoScript's built-in XSS or frame hijacking protection :) Bots will not notice, and users get a scary warning message and believe your ad is trying to hack them. Not good for your conversion rates, but a pretty sure way to tell them apart.
Unless you're targeting noscript users there's no way they account for a significant portion of those 80% of clicks.
What you say is valid, however no-script users are a small minority of internet users. I doubt their numbers would tip the results in either direction.
This could be a businessperson's interpretation of what happened.
Or they might have indeed reinvented it. Kids these days and their Heroku! Back in my day we'd process each HTTP request by hand. And we'd do it uphill both ways in the snow.
using awk.
Hey now, awk is simply lovely. It's ridiculous how popular it is to slight it these days.
Indeed.
Most of the parsing required for the logfile analysis I describe is written in awk, and goes through roughly 1 million lines of logfile in about 75s on a commodity 2Ghz Intel Xeon.
Pre-caching of host lookups is done via xargs and a suitably high '-j' value. Running on a pre-sorted and de-duped list of IPs, this takes another few minutes. Lookups are read into hash tables for faster processing in the main awk parser (avoids system calls, DNS latency, and especially negative result caching failures).
It's fast and simple. Not a 100% solution, but quick to add bits to over time as new needs arise.
How does 'using awk' count as a slight?
It was the proper add-on to the previous comment kids today don't learn awk. Becuase they have Perl, Ruby, and Python which are supersets of the unix command line tool set. Awk was the king in the years before Perl.
I never really learned to use cut, so I sitll use awk for one-liners the deal with columns in delimited-data text-files.
Sorry, I read it as a slight because it came right after "uphill both ways in the snow".
Which usually precedes "Get off my lawn!"
Anyway, if I want to take gratuitous pokes at a language, I usually choose XML; sometimes java.
Which, of course, I've just been informed is the "preferred" integration language for the project I'm presently working on. Sigh.
You had awk? Ha! Luxuries! Why when I was a young programmer we had to write the code in the snow with our pee, and a compiler was just a word for the pilot of the hovering dirigible that read the instructions and passed them to the ALU, which was another fellow with an abacus. They would wrap the re- sults around a rock, and drop it on my house when the program would exit. We had to walk uphill...
Using butterflies: http://xkcd.com/378/
With VB6 you can check IP Addresses.
Please guys. This is not Reddit.
Damn straight. Cheap jokes are not allowed.
And that is how to do a c-c-c-c-c-combo breaker on Hacker News.
This is besides the point.
Is it though? It seems they spent a lot of time throwing more and more javascript at the problem until they realized they couldn't figure out what was happening. If they had real logging from the beginning, it would have unconfused them much sooner.
They were taking orthogonal measurements to see if the pattern would appear over and over. Also it sounds like each successive measurement technique was at a lower level than the previous technique. Sounds pretty thorough. I'd love to see some actual data though.
How would you use the access log to determine if client-side javascript was being executed?
For the page in question, count total hits in the server access log and subtract total hits reported by the Javascript analytics software.
Check if it's even being fetched for a start. Assuming it's not inline.
Browser-cache my friend. You'll need to poll the server on page load (or even better in this case, after you load the ad)
Use a javascript function to poll the server for something.
thanks for coining the word "unconfused"
Access logs can't record if javascript is enabled. There are other things I would be interested in if I were doing this too, I'd check the browser's TZ, check if flash was enabled, etc. and do it across sessions.
Depends how far they went profiling their visitors, it clearly says in the article they were tracking if js was enabled, so yes, you'd need to roll something or another by hand. GA relies solely on js for example.
easy: < script > (new Image()).src = document.location.href + "Hey, i have javascript < / script >
And then compare in the log.
genius
The way I understand this is that they want to log who accesses their pages on FaceBook's servers.
I am not familiar with how things work on FaceBook, but I would think that FaceBook users do not have access to the logs that FaceBook's servers write.
you're not alone. I've heard similar stories from other businesses that have tried to advertise on FB. At first I thought they just didn't track properly, but it seems it's a widespread problem.
That escalated quickly!
And so it seems that Facebook could be proven to be the new groupon.
I would absolutely not be suprised if the hits were FB bots - however I'd expect they wouldn't be that directly stupid so as to have an Ad-bot net in house. Surely this is some service they outsource for plausible deniability.
Something they would have learned from their relationship handing all user data over to the NSA/CIA.
EDIT:
I'd like to make a prediction on how this could potentially play out:
1. FB ignores them. They have nothing to do with this issue and they know the bots are tainting traffic - but they can't/wont do anything about it because it boosts their revenues.
2. FB may or may not be involved in the ad-bots, but, they will drop their $ contingency on ad-rev for the name and payout these guys to shut them up. Their stock was improperly priced and taken a beating. They don't want this coming to light in a large way as it will have a negative affect on the perception of their business model. They hav to be kind of careful of the Barbara Streisand effect here too...
3. Many more people will reveal the same results with empirical testing and it will be revealed that FB earnings are 80% over inflated. Their stock will drop to the predicted real value of <$10... maybe hit that actual projected number of $7.00 - Zynga will BEG FB to purchase them as they are now a penny stock and they can't sustain.
We need to see some real numbers before making these kind of predictions. What was the volume of clicks that OP couldn't detect? 10, 1K, 1M? It makes a big difference.
Also given FB users' aversion to ads this whole issue could be explained by users just clicking the back button. Ad supported sites have trained users to do three things very well: install adblock, habitually ignore ads, quickly click away from any interstitial or unexpected ads like on Hulu.
Another note on the "back button": I recently discovered the awesome power of "three-finger swipe back button" on Macbook trackpad. So it's possible users can "click" back even faster than before. I can and before the swipe I was well-trained in the art of "CMD back arrow" :-)
Totally agree. However, I wanted to put my thoughts on this down... Surely we need more info and time to see what the actual truth is - but I just do not believe that FB is in any way some altruistic innocent/neutral entity.
What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.
Also, FB may not have much of a choice to comply with those agencies, or find themselves pissing off a lot of high level people by refusing to comply with government agencies. But outsourcing / tolerating a network of bots to vastly boost revenue is probably considered fraud, no?
> What user data stored on FB servers could be useful to intelligence agencies of that level? People brag about crimes they committed, IP addresses are logged and people may "like" jihad but criminals targeted by the NSA are surely not at that level of stupidity.
It's not what people brag about, it's the social graph itself that's useful. All the other things like photo-tagging are extra candy.
More info at http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/al... and http://nplusonemag.com/leave-your-cellphone-at-home . Sorry it's late out here, or I'd copy/paste the relevant quotes for you--look for the bits about data mining and machine learning, especially in the 2nd article, everything is a signal and they do have the computing power to process it all, apply Bayesian belief nets, algorithms can pull intelligence information about you from the local shape of your social graph that you don't even realize can be inferred from it (humans are not very good at reading complex info from big graphs much further than simple "friend-of-a-friend" relationships).
All this happening as employees in FB are itching to cash out their options in a few months.
So without ANY facts your conclusion is that Facebook is fraudulently deceiving advertisers.
And that they are doing this using experience from working with the NSA/CIA in order to cover up the fact that $800 million dollars of Facebook's yearly profits will disappear overnight once people realise the truth. And as such they are willing to pay out the ad-bot companies in order to hide this conspiracy.
Right.
I think problem is more deep, I do not want to make other Conspiracy Theory but here some story from other huge Russian social network (vk 150m. users). There is one small, not that big group (like fb groups), users made agreement, that no one will post next day (something like honeypot), and those who post would be considered as bot and baned forever. So next day couple or so bots generated comments encouraging others do not go on strike against some political party [0].
I do not know is it applies to FB, but those who have like 100k bots have some power too.
[0] http://imgur.com/Vvkbx screenshot in russian
Edit: spelling Edit2: proper link
>...without ANY facts...
What about the farking article? Where they proved that 80% of their clicks were from bots. Then several others confirm they ahve the same experience?
What about those facts?
I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.
Am I extremely distrustful of Facebook? Of course! Maybe you have not followed their track record or the character of their founder.
If you are into conspiracy theories then how about maybe this is Google directing a massive bot net to make facebook ads useless?
> I also speculated on what FB is doing. Them giving user data to the NSA/CIA - that is a supposed fact, sure, but I am not the only person who believes this.
Great argument! A lot of people believe it, so it must be true! Are you for real?!
It is not my responsibility to educate you on modern internet history. This has been argued here on HN prior; Do you recall the backdoors that AT&T put at 600 Folsom, in SF?
And the backdoors in Cisco's equipment that are a requirement by the federal government?
Maybe you thin this shit is "conspiracy theory" and if you do, you're simply a naive fool.
Look around you at what governments are doing. The NSA has records on everything you do online. They had a system in 2005 which could trace communications between users to 6 degrees, automatically.
It's odd to me that your posts aren't more downvoted than they are.
You think I make this shit up? There are some really naive and stupid people on HN that have zero grasp on the history of intelligence agencies.
So, this is why people don't take you seriously.
I know a couple of guys like you in my local activist community, who take a very hostile "I know the truth and you're all fools!" attitude, despite the fact that their audience is mostly very sympathetic to most of their assertions. We know spying on the Internet takes place; HN is full of cantankerous old Internet geeks who've seen, first-hand, plenty of examples of the state (whatever state you may choose, as it happens all over the world) behaving unethically on the Internet, spying on people and punishing people for things that shouldn't be crimes.
But, your paranoid approach is counter-productive. You might as well be working for the people you claim to be afraid of, for all the good you do (negative good; you're convincing people that the folks who believe the government is spying are all paranoid nutjobs who scream at anyone who has the gall to mention other possible explanations).
So, let's review:
1. Just because spying has taken place, and is currently taking place, and may even have the complicity of facebook in that spying, it does not mean that facebook is running a botnet to steal advertiser dollars. The simplest explanation is that facebook looks the other way while others run the botnets. facebook wins (a lot, as long as most advertisers don't know it's happening), botnet owner wins (a little), and the advertiser loses. But, there are other plausible explanations, including incompetence.
2. When you paint things in a "Either you accept my theory in its entirety, or you're all idiots", you force people to choose a side. Nobody wants to be on the same side as an asshole, so you force them to choose the other side. You make people who may even agree with you (to a greater or lesser degree) to begin to formulate plausible reasons for why you're wrong about the crazier stuff you're spouting...further convincing themselves that you're entirely wrong. The best you can hope for is people ignore you and don't have the chance to be inoculated against your ideas; having you as their first exposure to these concepts guarantees they will be less likely to believe them in the future, even if they come from a more credible source. Humans are funny creatures.
Thus, I would point out that there are some really naive and stupid people on HN that have zero grasp of effective argument, persuasion, and even basic logic.
You might be well-served by reading about non-violent communication: http://en.wikipedia.org/wiki/Nonviolent_Communication
Edit: Removed the word "schizophrenic" as it was an insensitive use of the term, and was counter-productive to making my point.
Hey, calling people "schizophrenic" is not cool. Just sayin'.
You're right. It was an unproductive method of describing the behavior I was seeing (and was too easy to interpret as saying the person is schizophrenic rather than exhibiting behavior I associate with schizophrenia). It was also insensitive to schizophrenics.
In my defense, schizophrenia runs in my family, and I'm very familiar with it...I don't think of it as an insult. But that's a local custom in my family that I shouldn't think follows in the rest of the world.
When describing behavior and not as a slur I don't see any problem.
>But, your paranoid schizophrenic approach is counter-productive.
Really, now I am a paranoid schizophrenic?
Just because I make claims which are readily confirmed and were completely available in the media - even the EFF filed suit on the AT&T events...
Yet, for some reason, it is my responsibility to educate everyone every single time someone new comes along who hasn't been following these things closely.
Now I am a paranoid schizophrenic?
Also, its a strawman to focus on my tone, rather than content. You're trying really hard to be overly pedantic and, frankly, an asshole.
Thanks for the link on communication.
What I am witnessing is someone trying to inform people, while effectively censoring himself by delivering in such a way as impedes its delivery.
You obviously have a vested interest in seeing your "message" be received, so stop ignoring what people are telling you about your tone and change it. Figure out what it is that people do listen to, or else you have no one but yourself to blame when they take issue with your tone.
To wit: Your tone is defensive. Pejoratives and cursing undermine your expression. What you are effectively telling people is that your thoughts are not important enough to merit self-restraint. Detailed explanations following assertions are also helpful.
FWIW, I work in the performance marketing industry and, IMO, the most complicity facebook could be said to have in this issue is in not placing a sufficiently high priority on preventing bots from clicking on their ads. Even if it is a sizable project there, they represent such a large target that the difficulty of the job becomes much greater. I see no salacious story here, other than a company found itself unable to optimize a campaign into profitability, which I think says more about them than facebook. Ho hum, find a different traffic source and move on.
Thanks - Ill take the constructive criticism on my tone.
However, I will point out that with respect to your comment on FB's complicit actions due to the daunting nature of the problem, this does not take into account their other actions of a 24K ransom on the domain.
Everyone can argue in any direction they want - but neither me nor anyone else is really going to know until we get further down this path...
He's not arguing with you, he can't be using a strawman. He's just pointing out that other people will be distracted by your tone, making them ignore your (mostly correct) message.
Thanks. In that case, I will try to be less emotional about this issue.
When I was talking about China hacking lockheed as far back as 2005, everyone said I was nuts! (I freaking worked at lockheed!)
There needs to be a better way to log and track this stuff so that we can point people to some sort of Tyranny Wiki.
And there are some that actually know what they're talking about in terms if intelligence agencies but choose not to indulge in idle conspiracy hypothesizing. The real operations aren't something that the people with actual knowledge would ever discuss. Those that do claim conspiracies are often ill informed or just have read too many spy novels.
>in all of our years of experience, only about 1-2% of people coming to us have JavaScript disabled //
I use noscript. When I enable it for visits it's usually by enabling the domain itself and leaving third-party scripts disabled. Depending on your area this could be significant.
Also the possibility of blocking of the scripts by other elements not loading - for example if the browser can't parallelise the requests for some reason and a preceding request can't be handled then the page might be displaying well before the script is loaded, especially if it's loaded at the bottom of the markup.
I think unless you run a forum for javascript virus injection techniques, you're not going to see 80% visitors with disabled javascript.
I didn't write an own analytic program but can confirm that 80% of clicks from my Facebook ads don't show up on my GoSquared dashboard.
And here is a screenshot: http://i.imgur.com/4hu66.png
What happened? Is reddit offline again or something?
There is (subjectively) a marked deterioration of comment quality on front-page discussions in the past few days, I feel. I have also noticed that HN articles are getting linked on reddit more frequently (I use both networks, but for highly disparate purposes - gotta get those funny .gif's somewhere). No data supporting this but I fell it's a trend and I'm not liking it.
Few days? Try since forever. As people increase the SNR gets larger. Even on reddit when it started, it was heavily tech focused, you'll find a comment that says it's going downhill. Hard to say what the numbers on HN are like growth wise in comparison to reddit (which has their subscriber numbers on display).
Here's ascii of the screenshot in case they censor your picture of text.
http://pastebin.com/raw.php?i=HHeY5nP1
Here's the MD5 of your ascii version of the screenshot of the blog post just in case: 14e34998a7bc46174bccd981ad5f41f6
I acknowledge receipt of the MD5 of the ASCII of the screenshot of the blog. Please acknowledge receipt of acknowledgement.
I give you a hamburger.
Alert! The MD5 hash of the ASCII screenshot I computed does not match yours:
$ curl -s http://pastebin.com/raw.php?i=HHeY5nP1 | md5sum 7b69269bd0e25daa8e8b171e5d513011
waiting, when screenshot will say "Mark Zuckerberg" likes this :-)
I don't understand why you would post that here. Can you explain?