Video web crawlers and the Google Monopoly over them.

Happy 2021 – Web crawling only gives you YouTube first, last, and always.

Back in 2016, we made an article about asking YouTube “Where is the fair use?” which was a slogan that the Nostalgia Critic was behind as parodies and criticism of film and literature is a form of free speech. Despite all of the Drama-Lama action that his agent “Channel Awesome” happened which is beyond the scope of this article. The statement was a valid question to ask YouTube. To the point where we even got hit with a copyright strike over a song “William Tell” which is a classical music song that falls underneath the category of public domain.

Read on if you want to know more.

betamax video tape - S-Config.ComVideo tutorial fallback mirrors:

In case you have no-script enabled or for some reason cannot see the title video on this website. We have provided direct links for these videos. For more information about the standards we use on this site click here if you would like to know more.

Disclaimer time.


We will try to keep this article as up-to-date as best as we can. However, since we are talking about search engines which is an ever-evolving technology you have to take the time this article was published which is January 01, 2021, at face value. If anything this posting should serve as a historical point within the internet as to how the normalcy of the net has become.

Humble beginnings.

Google Search Results for S-Config.ComWe were happy that Google was indexing our website and categorizing the hard work that we put into our videos and respectfully placed not only our site into the videos tab but also indexed those videos as we were also using later encoding technologies such as WebM. Although MP4 video encoding has gone the way of the dodo on this site Google continues to index our videos and respectfully place them into the “Videos” category of their search engine. It is our guess that their web-crawlers go through the content of each of our blog articles and upon finding a link to a self-hosted video another half of Google’s crawler goes to work indexing said video.

This meant that if someone types in this very website “s-config.com”. Our videos show up before YouTube. As it should because domain and text coming from that domain should be priority over a third-party site like Vimeo, YouTube, Etc. This was good news, because if more people self-host their videos then in turn those people become less dependent on YouTubes tyranny.

The other sites.

What about sites beyond the empire of Google? Sure, they do not have as much pull as Google does here in America. One can even make the argument that it’s virtually impossible to compete against Google. That Google in itself is a Monopoly. But really how far does that go? If other websites are using their own engines doing the same thing Google is doing then surely they may even provide a better result structure than Google itself, right?

Well! Let us find out:

Duration and environment.

Just for reference party people. My site has been online for 8 years with maybe an hour of downtime per entire year. It’s safe to say that thanks to AnubianHost my uptime has been about %99.8 and the %00.2 percent is mostly us being dumb-asses modifying code on a live production server instead of having a secondary backup WordPress site within our VPS. Since my site has been up 8 years with only 2 theme changes there really should be no excuse for any of the crawlers to do a good job at indexing this site. Hell, even the internet archive has a few of our pages.

The relevance of the “Video” tab.

I had some friends that legit said this to us.

I don’t even use the video tab on a search engine. I just go to YouTube. – Anonymous

Here-in lies the problem. Society is trained so well to go to one source for all of its information that they dare not try other methods. It’s a bit of a chicken-vs-egg mentality. How is a search engine supposed to get better at finding things if no one uses the search engine? Especially if the search engine decides that if your site isn’t Disney clean then you minus will be blacklisted like a porno service.

But looking at the egg. Could it be the quality of those methods which is why no one uses them? There could be the logic behind this statement. Let us find out.

The baseline comparison – Google at #1

S-Config Google search results.

Typing my domain name into Google does not change a whole lot. Because we know that google tracks the living hell out of you. We even used our Tor client and performed the same results in other countries just to ensure there was no favoritism at hand because we go to our own site so much. The articles shift a little under our site but the structure generally remains the same.

S-Config.Com video results.

Clicking on the tab for videos yields that Google is still doing a great job routing people to my site. That although the content shifted away from Ouya and more into gaming controllers it’s still pointing to videos that exist on my site.

S-Cofig.Com image results.

Google has been picking up more and more of our technical articles then art which just means we need to sit down and do more drawing to balance things out. Despite all of this Google stays focused on my site for a good page and a half before it starts looking at other web pages, social networks, and thumbnails. Which is very good and what we would expect if someone typed in our domain name.

Bing comparison test – #2

Bing pushed hard into the market in the mid-2000s powered by Microsoft as being a competitive search engine to Google. Bing claims to use its own methodologies of indexing and even has its own webmaster utility on ranking better with Bing. Looking through our own statistical analysis Bing only direct people to this site about %4 percent of the time in contrast to Google direction of traffic at %64.

S-Config.com - Bing Search Results

Bing search results start off similar to Google with a random selection of articles. Strangely the second result is yet-another-hit to my site referencing one of our LSD-driven short stories. and then it proceeds to reference other sites that we published works on and eventually forums and wiki’s there-after. Perhaps Bing likes stories more-so than Google?

 

S-Config.com - Video Results on Bing.The video search bar on Bing has to be the most disappointing out of all of the engines out there. Although if you search the forums of Bing; They claim that they index DailyMotion, Vimeo, YouTube, and even indexes videos on ABC, NBC, TMZ, The associated press.

Since we never made it to big-time news outlets (Nor would we want to) It gives this illusion that S-Config.com does not even exist. If you start clicking through some of these videos it’s YouTubers referencing to my site over one topic or another which is totally cool. However, Bing never indexes videos that exist on this very server. We have sent an inquiry about this on the Microsoft WebMaster Bing Forums however we are still awaiting some form of response to this.

This to us is super odd. Because Microsoft wants a piece of the market and wants to provide people with accurate and relevant searches for what they are looking for. But completely ignore the small-time webmaster despite offering tools for people like us to use. On one hand, you have Microsoft asking people to use their search engine. Only to turn around and divert all of the traffic to YouTube either out of incompetence, laziness or perhaps they were bought out by Google in this department. No one but the developers of Bing will truly know this answer.

Perhaps we talked some crap about how they performed planned obsolescence on Xbox 360 controllers. We doubt this as we’re rather sure Microsoft has some rather thick skin when it comes to criticism.

 

S-Config.com - Bing Image results.

Bings imaging results end up becoming increasingly more chaotic than Google. The third picture taking you off to a YouTube video named “Config” with someone from Asia appearing to be setting up their key-map controls for Monster Hunter. In essence, you start to see Bing’s engine fall apart by not honoring the quotations as Google would. However, on the more positive site, Bing tends to grab a lot more colorful articles we’ve done throughout our time instead of just screenshots. Which is nice.

Safesearch off.

Bing Video results for S-Config.com - SafeSearch off this time.

Apparently, Bing’s infinite wisdom has decided that my site is unsafe for children. That talking about Raspberry Pi’s, Xbox controllers, and android installation guides is the same as looking at hardcore porn. Brilliant.

It should also be noted that a lot of the videos bing is picking up a reference to older videos when we used to host MP4 files. Indicating that Microsoft is still holding onto dying codec standards of the .h264 as the only means of letting users within their search engine know that you host videos. We’ve asked the Bing Webmaster forums several times about how Bing processes videos and in general, we were met with absolute silence. Perhaps they’re too busy. Or, perhaps they simply do not care.

DuckDuckGo comparison test – #3

DuckDuckGo’s big deal is they are effectively Google but without the song from “The Police – Every breath you take” playing in the background while you type away. They do not spy on you. Because of this DuckDuckGo will not generate a ‘bubble’ of preferred searches based on your interests. Only show you the raw results.

S-Config.com - DuckDuckGo Search results.

DuckDuckGo’s search results mirror more of Bing than Google in terms of structure. Again, focusing on articles we’ve made before diving into forums and wiki’s.

 

S-Config.com - DuckDuckGo Videos results.

DuckDuckGo’s video search results are more disappointing than Bing in regards that for a company that is super-concerned about the privacy of its users it recommends going to a Sister company of Google that tracks everything you do. You could argue that depending on what you are searching for a YouTube link not being a priority is impossible. As we’ve stated with the Bing results. My videos are locally hosted. There’s absolutely no reason for other search engines to index/verify the videos that we post out there.

One last thing for those interested. we did turn SafeSearch off on DuckDuckGo as they tend to bury it into the settings panel of their search engine. Even after disabling SafeSearch, we did not show up still. Indicating that either DuckDuckGo is also looking for MP4 files but as we serve those anymore this could result in a total absence of results going to our domain.

Also, note that it found out twitch profile. Which that profile has absolutely nothing recorded in it. Yet, it’s showing in the videos tab which is inherently wrong.

S-Config.com - DuckDuckGo's imaging results.Again, the video results from DuckDuckGo have that eerie Microsoft Bing vibe to them. Which of course makes us wonder if DuckDuckGo is relying way to much on the same search engine mechanism that Bing is using as well,

Additional DuckDuckGo hilarity

DuckDuckGo Privacy Warnings.

It appears that AFTER you click on a YouTube video link, and then tab back the DuckDuckGo reminds you in the most passive aggressive way possible that it send you to a website by clicking on the text. YouTube does not admire your privacy as much as DuckDuckGo does. How about this DuckDuckGo. You link videos on my site which has a far better track record of privacy then YouTube and you don’t have to show warning like this. No? Oh, okay.

If you click on the images within DuckDuckGo only then does it show you the warning about Privacy in Advance.

Dogpile comparison test – #4

Whoah, we haven’t heard from this engine since the mid-2000’s. However,  they are still in the game and they too have a images and video tab for us to check out. Perhaps the underdog of search engines will surprise us here.

S-Config.com - DogPile Search

So it’s not as structure mark-up happy as the other three search engines. But for the most part it stays on target with my website first, then other places that we’ve published second. The site also wants to suggest meta-data that someone might be looking for the command line “sconfig.exe” from Microsoft instead of me. We kind of expected that too.

S-Config.com - Dogpile image searching.

Mmmmmm, my favorite way of viewing images is if we’re on a tablet despite the fact we are on a desktop. Kind of a weird vibe when it comes to displaying the images in a mosaic fashion. But regardless it’s similar more to Google with filling up the first page with images from my domain before moving on to other sites.

S-Config.com - Dogpile video results

Dogpile needs to some tender loving care in the CSS department and build themselves a responsive website that can handle dynamic resolutions. Despite this, the search results are identical to DuckDuckGo with misfiring the same Twitch profile as well as repeating the same videos just in thumbnail/description mode instead of just picture mode.

After turning SafeSearch off and content moderation off on DogPile, having it crash, reloading again. Once again nothing about my site. Disappointing.

Yahoo comparison test – #5

Yahoo has had a decline from their search engine almost to the point where it’s laughable to even use them as a search engine. There was a point during y2k that Yahoo felt almost unstoppable. We were even using them to plan to go on vacations. With endless security breaches and it’s a miracle the company is still alive. Yet, it’s still around because Yahoo wants to become a media/news site and less of a search engine. Since if you go into the search engine it has the same tabs as everyone else we’re giving them a try next here.

S-Config.com - Yahoo Search results

Hmmmm. The results certainly scream Bing on this one.

S-Config.com - Yahoo Image Results

The image results feels like this is what DogPile would look like if they actually fixed their site.

S-Config.com - Yahoo Video ResultsIf you replace the Yahoo logo with Bing No one would know the difference.

Yahoo Search Results of S-Config.com - SafeSearch Off.Again, with SafeSearch off. My older content that used to my MP4 appears on the listing.

Self-Assessment time.

Perhaps we’re doing something wrong? Perhaps we’re unanimously pissing off every search engine in existence? Well, we actually did find some things wrong with our site.

One of those things is the usage of sitemap-video.xml . Apparently, every plugin known to WordPress Claims to reference your Videos and catalog them into a sitemap-vimeo.xml for not only Google to catalog but for Bing and every other search engine. Sound amazing? Too good to be true? Well, it is!

Every single sitemap plugin we tried completely skipped over our self-hosted videos and only published a sitemap of YouTube links that we used purely for reference in some of my blogs. Great! So WordPress is flooded with plugins that shill for Google, YouTube and Vimeo, Fantastic.

To make matters worse. Because of my gallery system that we use for video presentation. The videos are linked to the gallery and not to the blog itself. So even if these plugins were working according to plan there’s no way in hell they would be able to associate what video goes to what blog entry.

After a bit of reading, we wrote our own sitemap-video.xml . It was a kind of a pain at first. But realistically since we publish a handful of videos it will be easy to maintain in the future.

I pointed all of the webmasters to take a look at it. They acknowledge that it’s a valid XML with 30 entries. But.. Alas, no change really.

Asking for help from the search engines themselves.

I first asked Bing Webmaster forum for support. The only response we received was from a person who was wondering what video hosting service I was using. Vimeo, Youtube. At which we responded with self-hosted which brought forth silence there-after.

The next I asked was DuckDuckGo which ironically uses Reddit which violates the privacy of its users as much as FaceBook. Learning from my mistakes on Bing we explained the situation in detail. and received nothing in return.

Apparently, the concept of self-hosting videos is alien and/or foreign to most understandably. But also bitterly sad at the same time.

Where do we go from here?!?

Now, a blog is supposed to help its readers. After all, if you’ve read this far you almost deserve a possible solution. Lights at the end of the tunnel as it were. To people like us, this is a lot of doom and gloom. But the reality comes down to how you feel politically about YouTube and the internet ecosystem that surrounds it.

For example, if your willing to be a bit of a conformist the best solution is to use them both! Have a website that you promote at the end of each of your videos with a blog entry going to your YouTube! However! Also encoding your own videos and attaching them to the blog as a backup like how we’re doing it!

Yes, you’re going to be exploited by YouTube like a person who just entered Hollywood. Yes, you might even get some of your videos demonetized or even pulled off of their network. But understand that if you have your own website with backups and you keep telling everyone about your personal site at the end of your videos and even on your profiles. They will come to see what was so terrible about your video that it was banned off of YouTube.

Also, if you preserve your MP4’s and spin the legal roulette wheel with MPEG-LA’s vague laws of usage. You might even have better luck having your encoded videos searchable on networks beyond Google.

Advice for our readers?

Finally, the last thing you could do (at least as an end-user/reader) is to at least begin taking away a search engines power. It’s ability to control you and your computer. To do this you have to make an informed decision that both doesn’t restrict your ability to find content or worse have content ignored completely due to big-techs censorship laws to appease communist countries.

To that end one solution is to try out presearch.org . Not even going to give you a referral ID because as a blogger concerned about privacy we want you the reader to think for yourself! We don’t make money on this ad and we’re only recommending these people because they seem more concerned with their blockchain then search manipulation. As we described in this blog entry there is indeed problems with Presearch but in contrast to the other big engines these problems appear to be due to a ‘work in progress’ . But right now at least to us this is probably the best solution.

Final thoughts.

We acknowledge that we are a small website. Thus, it would be very easy for the big search engines to overlook us hosting our own videos. Or, perhaps they simply do not want us small-timers ‘poisoning the well’ as it were.

Perhaps it’s only Google doing this with their videos section because you can’t think of Google as this monolithic organization. There are people in one project that have no idea what is happening in another. Such as the web-crawler programmers working on indexing video versus the marketing hacks at YouTube end.

When you start going from search engine to search engine. It becomes Evident. Unless you’re a major media outlet or Google, or you’ve been around since the dawn of time like Vimeo. you’re not getting any traction to your videos. Period. Independent hosting like what we are doing can’t even turn to projects like PeerTube because as discovered with Mastodon a decentralized network does not play nice with us a centralized network. In order to be acknowledged on PeerTube, we must join PeerTube and participate in their network. Which if it behaved more like Tor/Onion networks it would be acceptable.

Decentralized Fragmentation.

Unfortunately, that is not the case. As that too is subject to community-based censorship. Federated level fragmentation as discussed in this blog entry. So ultimately you will never see the whole picture with a service such as PeerTube. Perhaps that is also by design.

You find out that competing with YouTube is a lot like your local store competing against Wal-Mart. You can’t. They have more money than you, more clout than you, and will erase you out of existence in every way. Now, if blogging was like our livelihood we would be devastated at this point.

YouTube Stealing content monitization.

YouTube is very much aware that there is nowhere for a video blogger to go. This is why they decided to steal revenue from hard-working video producers. The very thing the music industry accused us of doing before we left YouTube. Total and complete hypocrites by nature so long as they get their ad-revenue.

YouTube is the criminals stealing with artists around the world!

Individuals have no powerful law-firm, no agents that represent them. Of course, I know this is what parliament/congress/senators are supposed to be there for. But with millions of dollars from Google funneled into their campaigns it’s easy for them to look the other way. That and most are so old they don’t even know what a computer is!

A giant middle finger to your copyright claim and if you don’t like it you CAN take your ball and well… Be erased from existence.

We have no rights, no permission over our own works when you submit anything to YouTube. If you go anywhere else on the net. Even the startup video hosting services. Prepare to be ignored by every major search engine and web-crawler in existence because for all we know every one of them Google makes sure that their service and ONLY their service appears on these engines. Face the void and be forgotten. That, my friends, is where the real monopoly is. You have to go out of your way to use another video service and even when you do you end up with minimal content.

We’ll face the void alright because throughout the course of my existence I have deprived YouTube of about 20 dollars. Now, the conversion of investment of site versus a loss of revenue is terrible. But it’s the little victories that we like.

Anyhow, for those who want to give it a go at self-hosting video blogging. Have the endurance of a God and balls that clink when you walk. Because it’s going to be a long haul. Until next time.

Server protect you.

+++ END OF LINE

 

2 thoughts on “Video web crawlers and the Google Monopoly over them.

  1. Hi there! I’m at work browsing your blog from my new
    iphone 3gs! Just wanted to say I love reading your blog and look
    forward to all your posts! Keep up the excellent work!

    Reply

Leave a Comment to the Void