Steem improvement proposal : Canonical links.
![](https://steemitimages.com/DQmNteDJkPoAxTAw9QbELWRztJQG6X1PEqLUpXF59iQGy5R/image.png)
![](https://steemitimages.com/640x0/https://steemitimages.com/DQmSE99h159zFWBo8tADHVVxfy6pMA9hFzy9wJBHk5P7TJN/image.png)
This is great! However; it can also be bad news for the steem ecosystem because of one thing :
When seaching for keywords on a search engine, it will look them up in their index. If they find the same content twice they have to make a choice of which one to display first ? One way to choose is by looking up the rankings.
For instance, if the same text appear both on wallstreetjournal.com and mysuperblog.com, it's unlikely that the article originated from mysuperblog.com. Since wallstreetjournal.com has a way higher ranking, it will show be shown first.
But with the steem blockchain the same content is displayed on a lot of various websites who are, in the end, front ends that display the steem blockchain. And this is where it gets tricky :
If I make a contribution on utopian, the text will be available on steemit.com and on utopian.io
So what will the search engine do ?
Show steemit.com even though I posted it on utopian and would like to keep the formatting of utopian.The same goes for other applications like steepshot and zappl.
As a result, it's very hard for dapps to do marketing because they are always in competition with steemit.com who make them less visible. This is the reason why you can't see articles when you go to utopian.io and you are not logged in.
This is even harder for small bloggers who want to post on their own blog and on steem to reach a broader audience and try to earn a secondary source of income (via SteemPress for instance). By using the steem blockchain they are basically hurting their own seo (search engine optimization). Although this sacrifice will be worth it for smaller bloggers, it may play out badly in the long run and is therefore a problem.
Which is a shame because it can be solved easily with what's called canonical links.
A proposed solution
Canonical links is a standard that allows to tell the search engines "ok I display this content but it originally came from this source"
In practice it's shown like so :
<link rel="canonical" href="http://original-article.com/article">
The issue is that this can only be set in the header part of the page. Which is basically a place where we have no control over.
So this is a proposition of a standard, it's heavily inspired by @jesta's work from last year.
Json metadata
I propose a new optional tag to the json metadata : "canonical"
The behaviour would be like so :
If the tag is not set, the front end (whatever it may be) won't add the line to its header. If however the canonical tag is set, then it'll add the provided url in the header.
The changes are fairly simple, and it would greatly help the whole ecosystem.
So for instance if steemit.com sees :
{
"author": "jesta",
"permlink": "test-post",
"category": "test",
"json_metadata": {
"canonical" : "http://mysuperblog.com/testpost"
}
}
Then in the header there will be
<link rel="canonical" href="http://mysuperblog.com/testpost">
I think there is a need to write the complete url in the canonical tag to allow anyone to add his own site regardless of how the url is shaped.
Canonical links for the popular dapps
This causes some overhead on the blockchain which will grow to be quite large if all the big dapps add a canonical link tag so I propose that if the "app" is recognized (aka it's part of a small list of whitelisted apps) then the front end automatically generate the canonical link :
{
"author": "jesta",
"permlink": "test-post",
"category": "test",
"json_metadata": {
"app": "busy/0.1"
}
}
From that the front end know that it orginated from busy, and thanks to the small list it can do busy -> https://busy.org and since almost all dapps uses this url scheme :
category/@username/permlink
it can easily recreate the cannonical link :
https://busy.org/test/@jesta/test-post
and add it in the header :
<link rel="canonical" href="https://busy.org/test/@jesta/test-post">
Which allows us to save a few bits on the blockchain.
Of course this means that there is a need for a list of the most popular dapps to be hardcoded on the front ends, but it's not like there are like 50 and other apps can still use the canonical tag for the same effect.
This is just an optimization of the space and is in no way a necessity and since it brings ongoing work like having to keep a list of all the popular apps and their urls, so the work may not be worth the saved bits.
Cascading links
What if stemit encounters a json_metadata that has the "app" and the "canonical" tag in json-metadata ?
{
"author": "jesta",
"permlink": "test-post",
"category": "test",
"json_metadata": {
"app": "busy/0.1",
"canonical": "http://mysuperblog.com/testpost"
}
}
We disregard the app tag and place the content of "canonical" on the link. So we cascade the links like so :
json_metadata.canonical
> json_metadata.app
> empty
So in the end, this i show it would work :
![](https://steemitimages.com/DQmTWPFyFM2XvMdYirinqrNgT8AQfyd6udNGfi6U6L1msKN/image.png)
I would be very interested in your thoughts on this, and if everyone agrees that this is the way to go, then I'll put the work into a pull request.
Update :
Created an issue in condenser with the proposal : https://github.com/steemit/condenser/issues/2505
In my view, this is one of the fundamental issues that really needs resolving for 3rd party app success.
A standard that everyone (including and most importantly, steemit.com) needs to follow.
I think too, and with steem success growing we'll see more and more dapps hence why I think it' so important to remove this problem that slows down adoption.
great catch @howo, was going to say this needs to be in Steemit GitHub submission & an Utopia post but with all the comments below, looks like this will be worked out the with all the dev here
just letting you know I'm following you now
I like the utility of the
canonical
property.The
app
tag is not particularly scaleable, and adds several brittle complexity obligations to all clients.I'd suggest we start with both a
canonical
and anapp_account
(that is an account associated with only the app that assisted in creating the post).At some point in the future we can either:
app_account
to publish a configuration set for in a special tx; so that all clients can generate the correct urls by pulling the most recent tx for that app_account.If we get sufficient community support, and add in the concept of trusted app_accounts (with some additional complexity) any of the above would get the job done 'well enough'.
Oh, I agree this really does need to happen. For all our sakes.
Nice to meet you @jesta
You are the best steemians
I think so
I agree with your approach 110%.
It will only help improve efficiency on the chain as well as avoid potential duplicate content issues. There are many On-Page SEO issues Steem has, many of the most critical I have outlined here...
https://steemit.com/steemit/@sf2/9-things-steemit-can-do-to-improve-on-page-seo-for-all-users
The more we can improve these technical SEO and accessibility issues, the more adoption and awareness we will create for Steem and the community.
Everybody wins.
What is the fundamental issue?
Great post, this was my initial concern from the very start. I love the fact that you've posted a problem and proposed solution. Keep it up!
This is very important point to consider even for WP integration. As when publishers are posting to their existing blogs that can also be pushed to the Steemit, before/if they move the the platform.
SteemPress could be really big in the next few years, this is the kind of Steemian global thinkthank collaboration and support we need to see from everyone in this global community so adoption speeds up benefitting all stake holders (Steem holders)
Actually there is lot of different url scheme now across apps:
https://steemit.com/{category}/@{username}/{permlink}
https://busy.org/@{username}/{permlink}
https://dmania.lol/post/{username}/{permlink}
etc...
It will be great if 3p apps know about others apps url scheme, it should not be a big deal to make a Gist within a JSON of every apps scheme so anyone can implement it.
I like this solution with both
canonial
andapp
, it give app more flexbility.Would it make sense for all apps to set
canonical
to their URL? This way n apps don't need to maintain n routes. Well, regardless, we still really need standardized libs for consistent rendering and maintaining any standards such as this.The issue with
canonical
IMO is that app canot change the link. If busy.org change his url scheme or busy.org become busy.com, we would still have old style canonical url from others apps. I would rather recommend 3p apps to use onlyapp
.For the standard that's something we can do at Steemit Inc. We can imagine a repo with all standards. Similar than what was proposed with SteemScript https://github.com/busyorg/steemscript. With that in place, 3p apps would have the possibility to PR change on the repo to update url scheme for example.
Busy is the dopest, words count and easy use makes it an ideal place to use steemit for me...Am your new biggest FAN!
The common solution when a website changes its url scheme, is to setup http redirects. So the work stays in the hands of the website owner to maintain their list of redirects.
This solution currently suffices, because search engines then update their index entries with the new url they got from the redirection, but I can see this solution not being prefect in the long term, since the canonical link entries in the blockchain can't get updated after a certain time. So either the website owner forever has to maintain an ever growing redirection table, or you just let the links break.
I commented above with a possible set of solutions..
https://steemit.com/steemdev/@howo/steem-improvement-proposal-canonical-links#@valuproposition/re-jesta-re-howo-steem-improvement-proposal-canonical-links-20180217t054225178z
@roadscape, I love the steemd you developed and always respect you.
However, I noticed you delegated SP to @s4s and the SP was used by @firepower to flag me just because I revealed the truth that he sent out up to 300 comments each day with a bot.
Furthermore, @firepower came to curse me and threaten me. He weaved lies and banned my steemit.chat account. Now he downvote every posts and comments of mine because I cannot agree a liar can be a witness.
To fool the innocent users, @firepower also followed more than 80,000 users. After some of the users followed back, he unfollowed all the users except 44.
I noticed you voted @firepower as witness. Therefore, I suggestion you to read the follow posts to know who is @firepower. Thanks
Yeah it is important that the steemit platform and all apps built on it unite together to promote the steemit platform instead of being in conflict with each other. I hope all parties involved agree to your proposal and make the required changes.
I never actually considered that this might be a problem, but now that you mention it, it's obviously an issue, and if it was to be fixed, I'm all for your proposed solution!
So although I agree with your solution, and I see the problem, I keep wondering if it will have negative side effects if it's solved.
One downside to actually solving this problem is that it might decrease Steemit's ranking, since it's basically competing with itself? I guess it's just a priority of what we want: A lot of decently ranked Steem-based sites or one really high ranking Steemit?
I also think it's important that people understand that all of these sites are connected in a collaborative ecosystem, and in that case I believe Steemit serves as a great portal to all those other apps. After all, posts written in other apps usually has a link to those apps, which I guess helps for their ranking?
In the end, I think Steemit will rank high enough anyways, so I'm all for implementing solution.
Yeah your remark is a good one, steemit basically allow lots of content to be found and it has been a great portal to the steem ecosystem, but now I believe it's slowing it down and it's time for steemit to let go and allow all of the apps to strive.
Like you mention, steemit.com ranking would get reset. But it's better to do this now than later when there are millions of people using it right? What do you think would be the immediate affect if the ranking dropped it, other than people writing about how steemit.com is shrinking in popularity?
There's nothing wrong with steemit losing some (or a lot) of its ranking over time. Steemit Inc's main purpose in life isn't to dominate the steemverse with steemit.com; it's to make the steemverse HUGE, diverse, and awesome.
Ned (the CEO) has said (approximately) this more than a few times times "If there are so many good products out there that people don't go to steemit.com, that's a measure of our success".
I was JUST talking to @themarkymark about this yesterday
Without a canonical fix we won't attract many of the big publishers, especially those with investors who will ask why Steemit outranks for their own content! :)
Yup. To make the STEEM token as attractive as possible, there should be as few cons as possible to using it.
This certainly is one which also has a relatively simple solution to it.
Agreed with both of your points. Ideally the platform becomes more attractive over time, not less.
SEO 101!
Interesting read, hopefully as it develops little kinks can be straightened out for the benefit of everyone.
yes it will, glad to see non-techies like you showing support
This is an excellent proposition. But if you come to think of it, at some point, it becomes outrageously uneasy to manage.
You made mention of the popular dapps being hardcoded. what happens when more dapps begin to originate, and then they become more popular, and consequently they feel they deserve to be top on that list instead of, say Busy.org. and let's assume they actually earned that spot better than busy. It becomes a battlefield of grief and uneasy rivalry.
Also, given that SMTs would be launched soon on the Steem blockchain, the number of these Steem frontends would increase drastically because more people would want to dive into the arena of building for this blockchain.
Consequently, you start seeing a massive influx of these dapps and any attempt to handpick the popular ones, no matter how objective and accurate they might be, there would always be an issue of supposed favouritism and inequality.
I think that is one of the main problems that might arise later, and it needs to be addressed before plunging right into this solution because trust me, then, it will become harder to manage and remain transparent.
If it has to be included at all, then it should be available for all dapps to make use of the canonical link. On the other hand however, this has adverse effects for th steem blockchain itself. All the attention and spotlight is taken from the default front-end, which I know for sure would hamper it from growing as far up the ranks as it should have.
I think the hard-code solution is fine for now when there are a handful, but it gets thrown out the window when the added size gets too large.
I think that what you talk about in the last sentence is actually healthy for steem and fine for all of us, if steemit is crap and no one wants to use it then let the market and the users decide which apps and sites become successful. I think it's actually an inevitability that steemit itself pretty much dies, considering how slow steemit's growth has been, how few users actually stay on the platform aside from the boom times, and the fact that historically blog and long-form platforms have all either ceased to exist or evolved into different models than the one steemit tries to emulate poorly.
Because of technical language I didn't get what was the crux of your post. I will try to read it again, to know how the suggestion of your 's will improve steemit.
One thing what I understood is the mentioning of source by using conical link. Please if you can, use some layman terms also to get people like me to understand. I am really applogitic if my opion is offensive.
Regards
On Second reading I got it. Yes your suggestions are great. The source must be linked because the third parties mush get their share. If the content is has some other sources, that must be highlighted.